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ABSTRACT 

This document describes six different approaches to 
outcomes assessments, approaches that are named in the titles of 
chapters 2 through 7. The chapters and authors are as follows: 
"Perspective* on Assessment Policy and Practice' 1 (Bragg, Harmon); 
"Total Quality Management" (Bragg); "Assessing Student Success" 
(Harmon); "Value-Added Assessment" (Harmon); "Concept Mapping" 
(Grayson); "Outcomes Assessment in Vocational Education" (McCaslin); 
and "Performance Assessment" (Mabry)^ A "Conclusion" following 
chapter 7 offers the following implementation strategies: (1) ensure 
that a clear institutional mission provides a framework for outcomes 
assessment; (2) focus assessment on institutional improvement; (3) 
gain top institutional commitment; (4) scan the external environment; 
(5) use democratic methods to conceptualize outcomes for vocational 
education, actively involving a wide range of stakeholders; (6) make 
the goal of identifying and meeting individual student needs a top 
priority; (7) build an organizational structure that supports 
data-driven decision making, planning, and program improvement; (8) 
use multiple measures to develop comprehensive records of student 
performance; (9) conduct routine program audits; (10) ensure feedback 
loops; (11) design reward systems that support the use of outcomes 
assessment system for improvement; and (12) develop dissemination 
strategies that support data use. Reference lists at the end of each 
chapter contain a total of 207 publications. (CML) 
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Introduction 



Over the last two decades, all levels of America's educational 
enterprise have come under increasingly stringent scrutiny. At 
least since the early 1980s, policymakers, business and industry 
leaders, and educational reformers have called for increased 
performance and accountability in all aspects of education. While 
calls for reform are certainly not without precedent, this most 
recent round of criticism is unusual in its protracted length, in the 
diversity of organizations that have issued reports and offered 
recommendations, and in the scope of the reforms that are being 
advocated. Efforts to set standard* at the national, state, and local 
levels are a central focus of many of these educational reform 
initiatives. 



State agencies, 
accreditation 
associations, and 
major employers 
press for 
outcomes 
assessment 



Postsecondary education has not escaped the calls for reform. 
According to Banta (1990), almost three-fourths of the states are 
recommending that their public postsecondary institutions make 
improvements in outcomes assessment. State governments and 
accreditation associations have been particularly influential in the 
movement to implement outcomes assessment in postsecondary 
education. Recommendations of these groups have focused on 
creating assessment systems that are sensitive to measuring 
educational processes — as they have historically — as well as 
educational outcomes. Postsecondary institutions are increasingly 
seeking to demonstrate that they are meeting reform requirements 
by improving educational processes, developing performance 
indicators, and conducting assessment to determine the quality of 
teaching and learning (Krcider & Walleri, 1988). 

Reform of vocational education at the postsecondary level is being 
encouraged by several groups. America's major employers are one 
group that is calling for better education to increase economic 
competitiveness. In discussing policy recommendations made by 
SCANS (Secretary's Commission on Achieving Necessary Skills) 
to improve workforce preparation systems, Brock explained that 
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Federal 

vocational 

education 

legislation 

mandates that 

state 

accountability 
systems measure 
academic skills 
and performance 



for SCANS to succeed, more businesses and schools must 
embrace these concepts. Our goal is to affect every school, 
every student, and every workplace. We hope to have a 
clear and specific impact on retraining programs and 
community colleges (1991, p. 25). 

Reinforcing the need for improvements in postsecondary 
education, current federal legislation in the Carl D. Perkins 
Vocational and Applied Technology Education Act of 1990 has as 
its major objective the development of improved accountability 
systems. This legislation requires that each state measure student 
learning gains in basic and more advanced academic skills and 
student performance in competency attainment, job or work skill 
attainment, retention or completion, or placement. The 
requirement to measure academic competence represents a 
departure from the type of outcomes traditionally measured in 
many states. Hoachlander and Rahn (1992) indicated that prior to 
passage of the federal legislation in 1990 only forty-one percent of 
the fifty states used academic achievement as a primary 
performance measure for postsecondary vocational education; job 
placement rates (86%) and occupational competency (50%) were 
more likely to be used. 

Given the scope and intensity of the educational reform movement, 
it is not surprising that institutions are being encouraged to look at 
a broader set of outcomes than had been used traditionally and to 
reexamine the means by which these outcomes were being 
measured. As new educational goals emerge to address reform 
directives and as new approaches to vocational education are 
employed, alternative approaches to outcomes assessment are 
essential. It is this need for alternative approaches that has 
provided the impetus for this book. 



A Conceptual Framework for Exploring 
Outcomes Assessment 



Terms used in 
this book 



The terms "outcomes," "assessment," "traditional," and 
"alternative" are used extensively throughout this book and applied 
to educational institutions, programs, and students. The term 
"outcomes" is used to refer to observable and quantifiable 
expressions of results that can accrue from the provision of 
education. The term "assessment" relates to gathering and 
analyzing information about how education can be improved and 
the impact that it has (Sims, 1992). Therefore, "outcomes 
assessment" refers to evaluative processes that determine the 
results of education at the institutional, program, and student 
levels. 
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Calls for reform 
include the need 
for improved 
assessment at the 
institutional, 
program, and 
student levels 



Wc believe that it is possible to look at outcomes assessment — its 
focus as well as its methodologies— from either a "traditional" or 
••alternative" perspective. We define "traditional" as those 
outcomes that have been routinely used in the practice of 
postsecondary vocational education, such as job placement, 
occupational competence, program completion or retention, and 
earnings. These outcomes have traditionally been measured with 
licensure and certification tests, locally-designed tests, follow-up 
surveys, and self reports* 

"Alternative" outcomes and outcome measures, cn the other hand, 
refer to results that have not been used extensively by 
postsecondary vocational education. Examples of outcomes that 
fall into this category include transfer success, job satisfaction, and 
individual attainment of educational and career goals. "Alternative 
assessment" approaches can refer to "virtually any form of 
evaluation other than traditional paper and pencil, machine-scored, 
or multiple-choice tests" (Stefonek, 1991, p. 1). The goal of 
alternative assessment is to produce more valid information about 
student knowledge and skills than has been available with more 
standardized testing procedures. 

We hypothesize that there are three types of outcomes to be 
considered when assessing postsecondary vocational education: 
institutional outcomes, program outcomes, and student outcomes. 
Within the range of educational activities, there are certain goals 
that can only be effectively and efficiently addressed at the 
institutional level. In some cases these are outcomes that represent 
an aggregation of all efforts occurring -cross the institution, such 
as institution- wide retention rates and student satisfaction. In other 
cases, the outcomes reflect the results of centralized efforts, such as 
the impact of economic development initiatives or the utility of 
management information systems. Often, outcomes assessment at 
this level is viewed as part of institutional effectiveness evaluation 
(Alfred & Kreider, 1991; Nichols, 1991; Seybert, 1990). 

Program outcomes, also referred to as functional-area outcomes 
(Alfred & Kreider, 1991), can serve as a bridge between 
institutional and student outcomes. They provide a means by 
which an institution and its stakeholders can measure the 
appropriateness and effectiveness of specific programmatic efforts. 
In some cases, programs may be assessed by aggregating student 
outcome measures. Traditional aggregated outcome measures 
include student enrollment, retention, graduation, and job 
placement rates and employer and graduate satisfaction levels. In 
some cases, however, outcomes must be assessed at the program 
level because that is where they have the most meaning. Often 
these outcome measures are linked to the efficiency with which 
programs use resources and personnel and to the effectiveness of 
services for diverse student populations. 



Finally, student outcomes are concerned with changes that occur in 
individuals as a result of their participation in an educational 
experience. The concept of student outcomes is frequently 
expressed in terms of the value that is added to an individual as a 
consequence of that participation. These outcomes include the 
knowledge, skill, and attitudes that learners are expected to acquire 
and demonstrate. They are measured by assessing an individual's 
performance. These measures include gains in knowledge and skill 
competence and arc frequently measured by competency- or 
performance-based tests (taking many forms) and follow-up 
surveys. 



About This Book 



Innovations in 
outcomes 
assessment are 
the focus of this 
book 



This book discusses innovations in outcomes assessment for 
postsecondary education, especially vocational education, that have 
been identified during a two-year research effort of the National 
Center for Research in Vocational Education (NCRVE) site at the 
University of Illinois. Our research has involved a wide range of 
two-year public and private community and technical colleges 
across the United States where we have had the opportunity to 
study and learn how assessment approaches have been employed. 
Although it is impossible to fully report the breadth and depth of 
our research findings, we have attempted to share the most 
important findings to assist you in pursuing future outcomes 
assessment endeavors. Ideas reported here represent what we 
believe are some of the newest and most promising ideas in how to 
conceptualize and conduct outcomes assessment in two-year 
postsecondary educational institutions. 



Intended Audiences 



Local leaders, There are two audiences of readers for this book. First, the primary 
institutional audience is comprised of local postsecondary educational leaders 

researchers, and who have responsibility for conceptualizing and administering 
policy makers outcomes assessment, especially for vocational education. This 
are primary group includes community college presidents, institutional 

audiences for this researchers, and vocational education administrators who have 
book primary responsibility for improving programs and meeting 

accountability requirements. For this audience, our goal is to bring 
a new level of understanding about outcomes assessment and to 
encourage the exploration of various approaches to measuring 
outcomes. Of course, full-scale implementation of any assessment 
approach will require more information than can be provided in 
this book, so we encourage readers to explore the extensive list of 
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references provided at the end of each chapter as a basis for further 
implementation efforts. 

The second intended audience includes federal, state, and local 
policymakers, government agency staff, program evaluators, 
researchers, teacher educators, and others who are involved in 
advocating, developing, or disseminating innovations in outcomes 
assessment for postsecondary vocational education. Similarly to 
the first group of readers, our goal has been to create a greater level 
of understanding about alternative approaches to outcomes 
assessment. To encourage further innovation for the purposes of 
improving practice is a second goal of ours for this group of 
readers. 



Chapters and Content 



Our primary purpose for preparing this publication was to describe 
various outcomes assessment approaches, to challenge you to 
explore them further, and to encourage you to consider how they 
can be applied in your institution. The chapters are sequenced to 
address institutional outcomes, then program outcomes, and finally 
student outcomes assessment Each chapter provides a discussion 
of each outcomes assessment approach and suggests 
implementation issues and strategies. Most chapters also discuss 
how the particular outcomes assessment approach has been applied 
in one or more postsecondary settings. 

To provide a basis for exploring alternative approaches in 
outcomes assessment, Chapter One discusses external forces on 
accountability, particularly those stemming from recent policy 
recommendations and legislative mandates. Then, internal forces, 
which largely reflect shifts in institutional missions and student 
populations, are presented. With this background, research 
findings regarding outcomes assessment practices at selected two* 
year postsecondary institutions are presented. The chapter 
concludes with a discussion of challenges and recommendations 
for implementing outcomes assessment in postsecondary settings. 

Chapter Two advances the argument that the "M" in TQM (total 
quality management) could just as easily stand for "measurement" 
as "management." This Chapter discusses the philosophical and 
theoretical bases for TQM and describes how this comprehensive 
approach can be implemented to enhance institutional effectiveness 
and ensure beneficial institutional outcomes for students. 
Observations are made about how TQM is being implemented at 
Fox Valley Technical College in Wisconsin. 
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Chapter Three focuses on an institutional approach known as the 
student success model that parallels, at least philosophically, many 
of the tenets of TQM, While some readers may associate this 
model with a counseling and educational planning model, we 
believe that it can provide the basis for outcomes assessment as 
well. This Chapter explores the way Santa Fe Community College 
(SFCC) in New Mexico has applied the student success model to 
develop and maintain institutional effectiveness and outcomes 
assessment 

In Chapter Four, the concept of value added is applied to outcomes 
assessment and discussed in relation to institution, program, and 
student outcomes assessment The value-added assessment 
approaches used by Delaware Technical Community College, 
Northeast Missouri State University, and Alverno College are 
discussed. Issues surrounding the relationship between value- 
added and outcomes assessment are presented. In addition, the 
controversy about using pretest and posttest designs for measuring 
value added are explored and an argument for using performance 
assessment is made. 

Chapter Five describes a recent innovation that offers promise for 
getting program-level outcomes assessment efforts started, no 
matter whether they are based on a TQM, student success, value- 
added, or other approach. Referred to as concept mapping, this • 
innovation is a bottom-up approach for producing a picture of how 
educational programs work and how they produce outcomes. This 
chapter illustrates how concept mapping was used to identify 
outcomes for vocational education at Black Hawk College in 
Illinois. 

In Chapter Six, a schema for assessing educational, economic, and 
psycho-social outcomes for vocational education programs is 
presented. This chapter suggests that these outcomes must be 
considered in light of the educational needs and processes that 
influence vocational education. To conclude the chapter, a five- 
step process for conducting outcomes assessment is presented. 

Chapter Seven addresses the contemporary topic of performance 
assessment of student knowledge and skills. This chapter defines 
common terms such as "performance assessment," "alternative 
assessment," and "authentic assessment" The chapter describes 
assessment techniques such as profiles and records of achievement, 
portfolios, and performance tasks. The chapter also discusses how 
these assessment techniques are being implemented in several 
secondary and postsecondary organizations and describes issues 
surrounding their use in these settings. 

Finally, the Conclusion summarizes the main ideas in the book & d 
offers some suggestions for implementing outcomes assessment in 
postsecondary vocational education. This chapter reiterates our 
challenge to consider alternative approaches as a means of 



improving outcomes assessment practices for postsecondary 
vocational education. 



How to Use the Book 



Use this book to 
generate 
discussion, 
stimulate new 
ideas, and 
further 

implementation 
of outcomes 
assessment 



Clearly, this book does not provide a step-by-step, cookbook 
approach to outcomes assessment That was never our goal! 
Rather, we intended the ideas presented here to generate discussion 
and stimulate thinking about how alternative approaches to 
outcomes assessment could be employed by postsecondary 
institutions. Sometimes we share concepts that are not traditionally 
viewed as being a part of outcomes assessment strategies and 
techniques, especially for vocational education. For example, 
TQM, value-added, and concept mapping may seem out of place in 
this book. However, we believe that these innovations may 
actually offer the most promise for addressing the challenges 
postsecondary educators face in creating assessment systems. 
They may be particularly helpful in creating a holistic and 
systematic approach to assessment for postsecondary vocational 
education. 



Indeed, throughout this book, we encourage readers to think 
beyond traditional approaches to outcomes assessment — as we 
have attempted to do — to consider new ways to meet an 
institution's accountability requirements. We believe Kouzes and 
Posner said it best when they wrote, 

The search for opportunities beyond tradition is an 
exploration in the new. It requires individual creativity and 
organizational innovation. That means that a leader must 
have an openness to ideas and a willingness to listen. It 
means that leaders must try untested approaches and accept 
the risks that accompany all experiments (1990, p. 54). 

By challenging your thinking about outcomes assessment and 
encouraging you to consider how alternative approaches could be 
implemented in your institution, we believe the opportunity to 
advance the practice of outcomes assessment — to "explore the 
new" — can lead to innovation and improved education for all. 
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Chapter 1 

Perspectives on Assessment 
Policy & Practice 

Debra D. Bragg and C. Michael Harmon 



O ver view The current debate about outcomes assessment and accountability is 

largely a result of recommendations stemming from educational 
policy, research, and practice* Together, shifts in these areas have 
increased demands on educators to assess the results of their efforts 
(Ewell, 1989). Throughout this document, outcomes are viewed as 
the consequences of involvement in or with education. 

This chapter is designed to provide the context for later chapters 
which define and describe evolving assessment approaches that can 
be used to address current demands for measuring outcomes and 
demonstrating accountability. Discussions about each of these 
approaches refer directly to postsecondary education, especially 
vocational education, in an attempt to advance implementation of 
outcomes assessment within that arena. 

In this chapter This chapter focuses discussion on the following topics: 



Topic See Page 

Policy and Legislation 2 
Other External Forces on Accountability 3 
Internal Forces on Accountability 5 
Current Outcomes Assessment Practices 6 
Challenges for Practitioners 9 



Policy & Legislation 



Recent reports 
call for reform 



In recent years, federal and state governments, the National 
Governor's Association, professional education associations, 
businesses, and other private groups have called for a wide range of 
reforms* Increasingly, these groups have encouraged educators to 
look beyond traditional practice to find ideas for improvements. 
Linkages between educational systems and the workplace are 
particularly prevalent in these reports. In addition, calls for 
improvement in assessment and accountability are plentiful. 
Selected prominent reports, listed chronologically, follow. 



Report 



Contents 



Workforce 2000: Work and Workers 
for the 21st Century (Johnston & 
Packer, 1987) 

The Forgotten Half: Non-College 
Youth in America (William T. Grant 
Commission, 1988) 

Labor Force 2000: Corporate America 
Responds (Allstate Forum on Public 
Issues, 1989) 

America's Choice: High Skills or 
Low Wages! (Commission on Skills 
of the American Workforce, 1990) 

America 2000: An Education Strategy 
(U.S. Department of Education, 
1991) 

What Work Requires of Schools: A 
SCANS Report for America 2000 
(Secretary's Commission on 
Achieving Necessary Skills, 1991) 

Education Counts (Special Study 
Panel on Education Indicators, 1991) 

Learning a Living: A Blueprint for 
High Performance (Secretary's 
Commission on Achieving Necessary 
Skills, 1992) 



Discusses the serious mismatch 
between the skills of the workforce 
and jobs of the future. 

Reports the problems of non 
college-bound youth in transitioning 
from school to family and work. 

Provides a business perspective on 
how corporations can contribute to 
educational reform. 

Examines problems with economic 
productivity and advocates higher 
standards and far-reaching reform. 

Provides a strategy for meeting six 
National Education Goals by the 
year 2000. 

Examines the demands of the work- 
place and identifies five competency 
and three foundational areas 
required to enter employment. 

Recommends an information system 
and education indicators on issues. 

Proposes a learning system 

designed to change schools, 

workplaces, and assessment 
systems. 
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Each of these reports contributes to the notion that all levels of 
education in America must change to support individual wage 
earners, businesses, and die nation's economy* For example, 
America 2000 proposed a set of six national goals designed to drive 
educational reform. The fifth goal has direct implications for 
postsecondaiy education It states that "every adult American will 
be literate and will po&o$s the knowledge and skills necessary to 
compete in a global economy and exercise the rights and 
responsibilities of citizenship" (U.S. Department of Education, 
1991, p. 19). 

Current federal legislation for vocational education has reinforced 
the importance of education for all aspects of employment as well as 
the need for accountability systems to measure their effects* Title I, 
Part B, of the Carl D. Perkins Vocational and Applied Technology 
Education Act of 1990, simply Perkins II as the law is commonly 
known, requires that each state measure student gains in academic 
skills, and occupational performance (Section 115(a) and (b». 
It also encourages states to effectively serve special populations 
(Section 118(a)) and coordinate efforts among education and 
training programs receiving federal funds (Section 1 15(c)). 

Measures of these gains are either being contemplated or made by 
states (Hoachlander & Rahn, 1992). In reviewing these outcome 
measures, it becomes apparent that many can be described as 
traditional in the sense that they continue to reflect the types of 
outcome measures historically mandated by vocational education 
legislation and government agencies or that they have been used 
routinely in the practice of vocational education. (See, for example, 
Darcy (1979), Stern (1988), and Wending & Barnard (1985)^ 

New initiatives in the Perkins II legislation create a heightened need 
for outcomes assessment For example, Title III, The Tech Prep 
Education Act, provides funding for new Tech Prep (technical 
preparation) educational programs to articulate secondary and 
postsecondary curriculum. In addition, the need for applied and 
integrated academic and vocational education curriculum is 
emphasized in the Tech Prep Act and elsewhere in the legislation. In 
order to satisfy these initiatives, outcomes assessment that can 
provide information for program improvement as well as determine 
the impact of these new initiatives on institutions, programs, or 
students will be needed. 



Other External Forces on Accountability 



Another important force in educational accountability has been state 
government. Many states have been active in promulgating 
educational policy related to postsecondary outcomes assessment 
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Indeed, "almost three-fourths of the states are asking public 
institutions to do something about assessment" (Banta, 1990, p. 3). 
As with testing related to Perkins II, the perceived need to "do 
something" has frequendy resulted in die consideration or adoption 
of standardized tests. Recently, however, it appears this emphasis 
may be diminishing as states seek to attain more comprehensive data 
on a wider range of outcomes for their postsecondary education 
institutions (Ewell, 1992). 

Postsecondary education in the state of Washington is one instance 
in which standardized testing of all students at the end of the 
sophomore year has been considered and rejected. In 1989, the 
Washington State Higher Education Coordinating Board chose to 
not require a statewide testing program, preferring instead to fund 
faculty-driven, institution-centered assessment development and 
implementation efforts at each public community college and four- 
year institution (Moore, 1992). The result has been the development 
of course- and program-level assessment approaches specifically 
designed to meet the needs of individual colleges and their 
stakeholders. In some cases these approaches include portfolios, 
performance assessments, narrative assessments, and alternative 
assessment approaches. 

In addition to issues related to testing, concerns continue to be 
expressed by students and parents about the rising cost of a college 
education, especially in light of the country's economic difficulties 
(McGuinness, 1991). At the same time, many state leaders see that 
their postsecondary education systems are overextended and 
underfinanced. They show concern about whether institutions can 
deal effectively "with other states for new industry and jobs if we 
keep spreading limited resources across too many mediocre 
institutions" (McGuinness, 1991, p. 1). The states were viewed by 
organizations that advocate reform, for example the National 
Governor's Association and Education Commission of the States, as 
the appropriate agent of change. This perspective has reinforced 
state mandates for performance-oriented accountability reports in 
addition to assessments of student learning. 



State 

governments 
seek 

comprehensive 
data on a wide 
range of 
outcomes 



Accrediting In response to the U.S. Department of Education and the Council on 

agencies require Postsecondary Accreditation (COPA) in the late 1980s, the six 
institutions regional accrediting agencies expanded their emphasis on outcomes 

attend to assessment. According to Banta (1990, p. 3), "This may have been 

outcomes the most influential development in capturing the attention of private 

assessment colleges and universities with respect to outcomes assessment/ 

issues Today, all six of the regional agencies require that their member 

..institutions attend to outcomes assessment issues in some v ay 
(Marchesc, 1990). In some cases (the New England Commission 
on Vocational Technical Career Institutions, for example), a 
mission-based approach was chosen. In other instances, exemplified 
by the North Central, Southern, and Western associations, an 
effectiveness indicator-based approach matched to the institution's 
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mission statement was chosen. Among the associations using an 
effectiveness indicator-based approach there appears to be 
considerable variation in the weight put on mission statements (high 
in the Southern and North Central associations, for example) and 
effectiveness indicators (high in the Western association). As a 
result of these efforts, by 1988 all of the accrediting associations 
"required evidence of achievement as a part of their institutional or 
programmatic reviews" (Marchesc, 1990, p.371). 



Internal Forces on Accountability 



A redefinition 

of outcomes 

reflects 

increased 

student 

diversity 



Of course, not all outcomes assessment programs are the 
consequence of external pressure. Before federal and state 
government bodies and accrediting agencies began stressing the 
importance of assessment, ground-breaking programs were being 
implemented at several two-year individual colleges such as Johnson 
County Community College in Kansas, Miami-Dade Community 
College in Florida, Midlands Technical College in South Carolina, 
Mt. Hood Community College in Oregon, and Prince Georges 
Community College in Maryland (Alfred, 1991). Four-year 
institutions frequently identified as leaders in the field are Alverno 
College and Northeast Missouri State University (Banta, 1990). 

As a consequence of these and other calls for change and guided by 
the examples set at pioneering institutions, educators across the 
country have begun to reconsider the adequacy of both the goals that 
they are seeking to achieve and the mechanisms used to assess 
attainment of these goals in the form of outcomes. These changes 
have occurred in the areas of institutional effectiveness, program 
improvement, and student success. Postsecondary institutions are 
shifting away from measuring inputs to determine the quality of their 
services to measuring outcomes such as learner knowledge and skill 
attainment (Kreider & Walleri, 1988; Sims, 1992). These 
measurements are occurring because of the use of models that 
emphasize to varying degrees the evaluation of institutional mission 
and policies, programs or functional areas, and teaching and student 
learning (Alfred & Kreider, 1991; Nichols, 1991; Seybert, 1990). 

The need to redefine appropriate outcomes and outcome measures 
is, in part, based on the policy changes that are being called for. It 
is also based on a shift in the type of students being served by 
postsecondary institutions in general and by their vocational 
education components in particular. This shift includes increasing 
numbers of women and minority students. The percentage of part- 
time stadents of both sexes and all races is also increasing 
dramatically (El-Khawas, Carter & Ottinger, 1988). As a 
consequence, patterns of attendance are extremely varied, making 
them difficult to define. According to Adelman (1992, p. 4), the 
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challenge before community colleges is in understanding that "who 
attends cannot be divorced from an understanding of how they 
attend/ 1 As will be discussed later in this chapter, the diverse nature 
of postsecondary students and their related attendance patterns must 
directly impact strategies undertaken by two-year postsecondary 
institutions to conduct outcomes assessment (Ewell, 1992). 



Current Outcomes Assessment Practices 



During 1991, we conducted research to provide an understanding of 
the nature of outcomes assessment practices carried out by two-year 
postsecondary educational institutions in the United States or their 
administrative agencies. Using a purposive sample of fifty-four 
organizations identified by a panel of national experts to be 
outstanding in some aspect of their outcomes assessment process, a 
one-hour semi-structured telephone interview was conducted. 
Respondents chosen for the interview were usually institutional 
researchers because of their knowledge about assessment efforts 
occurring across the curriculum. Of the total sample, seventy-four 
percent were community colleges, thirteen percent were public 
technical colleges, two percent were private proprietary colleges, 
and nine percent were administrative agencies. 

The goal of this research was to address four general questions: (1) 
What outcomes are being assessed? (2) How are they being 
measured? (3) How are the results being used? and (4) What 
changes have occurred as a result of current outcomes assessment 
practices? These questions focused on general outcomes assessment 
processes as well as those specifically related to vocational 
education. An analysis of the findings revealed several important 
observations about practices conducted by institutions that some 
experts considered to be leaders in outcomes assessment. 

First, seventy-eight percent of the organizations reported that they 
were conducting outcomes assessment processes as part of an 
institution-wide effort. Of the remaining twenty-two percent, 
outcomes assessment focused solely on vocational education; 
however, it was noted that thirty-three percent of these organizations 
were exclusively vocational instructional facilities. Within the 
comprehensive community colleges that made up the vast majority 
of two-year institutions in this study, outcomes assessment 
processes typically included vocational and other general or transfer 
education curricula. Therefore, for the selected two-year community 
colleges in this study, outcomes assessment was viewed as a 
process inclusive of the broad spectrum of curricula across the 
college and not likely to be segregated to vocational education. This 
finding has direct implications for the development of assessment 
processes that include outcome measures that can be interrelated to 
determine institutional, program, and student success. 



Second, respondents were asked to describe the types of outcomes 
they wore measuring as well as their use of alternative outcomes 
assessment (e.g., performance assessment, portfolios, and capstone 
projects). Information obtained in response to these questions were 
widely varied, potentially indicating a lack of consensus about the 
types of outcomes that two-year postsecondary institutions should 
be measuring* However, given this variation in response, it is 
important to note that the outcomes reported seemed to us to be 
fairly typical of those two-year institutions have traditionally 
measured. Only a few of the institutions seemed to be assessing 
outcomes not previously within the repertoire of their institutional 
research efforts; still fewer were developing or implementing 
alternative assessment approaches for these outcomes. 



Outcomes 


Frequency 


Outcomes Assessment Methods 


Academic achievement 


13 


standardized tests, exit tests, 
portfolios, capstone projects, 
exams graded by panels of experts, 
faculty and student surveys 


Transfer success 


13 


transcript analysis, 

state tracking systems, 

feedback from receiving institutions 


Student satisfaction 


11 


student surveys 


Employer satisfaction 


10 


employer surveys 


Attainment of student goals 


7 


student surveys, 
tracking systems 


Employment success 


6 


state unemployment records, 
college placement office data 


Occupational skill attainment 


6 


performance and narrative 
assessments, licensure and 
certification exams 


Job placement 


4 


follow-up surveys 


Student persistence 


4 


enrollment patterns 



Third, respondents were asked to identify the driving force(s) 
behind any changes in their college's outcomes assessment 
processes over the past five years. Again, the responses varied 
widely; hovever, three forces were seen as providing the impetus 
for changes in a number of institution's outcomes assessment 
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Outcomes 
assessment 
provides 
information for 
institutional 
accountability 
and program 
improvement 



practices. The first force was the chief executive officer (CEO), 
according to twenty-two percent of the respondents. These 
individuals were seen as committed to using information obtained 
through outcomes assessment for decision making about the 
institution and for measuring student performance. A second force 
identified by the same percentage of organizations was state or 
accrediting agencies. This is not surprising given the importance of 
accrediting agencies to the entire outcomes assessment movement 
The third driving force behind changes was federal legislation, 
according to nine percent of the respondents. Similarly to 
accrediting agencies, legislation has mandated the development of 
outcomes assessment processes for postsecondary institutions. 

Conspicuously absent from this list was driving forces related to 
student or faculty needs as well as program or curriculum 
improvement, which could lead one to conclude that institutions 
viewed outcomes assessment as an accountability rather than a 
program improvement tool. When respondents were asked how 
outcomes assessment information was used, however, thirty-three 
percent indicated program evaluation, thirty-one percent reported 
strategic planning, twenty-six percent described curriculum 
development, and eighteen percent spoke about resource allocation. 
Some of these processes tend to link information obtained through 
outcomes assessment back to institutional and program functions 
and are likely to contribute to periodic review and improvement. 
This type of information seems to be critical to making 
improvements in programs. This finding emphasizes the need to 
develop outcomes assessment processes that can address a broad 
array of needs, especially accountability and program improvement 
needs. 

Finally, respondents were asked how outcomes assessment had 
changed their institutions. Nearly all respondents reinforced the 
notion that outcomes assessment was impacting institution-wide 
processes such as administration, personnel development, and 
curriculum development. Although responses were widely 
disparate, improvements in institutional decision making and 
planning were identified by thirty percent of the respondents. Other 
responses were better staff development, better alignment with 
business needs, improved climate for on-going assessment, more 
faculty use of results, and more confidence in institutional 
effectiveness — each identified by less than eight percent of the 
respondents. 

To summarize major findings from this study, the potential exists 
for utilization of outcomes assessment to provide information to 
address institutional accountability as well as program improvement. 
This study indicated that postsecondary educational organizations, 
viewed as leaders in outcomes assessment, measure a wide variety 
of outcomes through relatively traditional means. Whether 
stimulated by internal or external forces, they conduct outcomes 
assessment in a manner that impacts the entire curriculum and 
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potentially yields improvements in administrative, instructional, 
personnel, and collaborative functions. Of importance for future 
work, the identification and development of outcome measures, 
beyond those traditionally used in postsecondary education, await 
development and implementation. 



Challenges for Practitioners 



Demands for 
outcomes 
assessment can 
be a challenge 
to practitioners 



As the demands and expectations for outcomes assessment increase, 
so do the challenges for practitioners in postsecondary educational 
institutions. Ewell (1992) has identified several challenges that 
surround the multifaceted, pluralistic environment of these 
institutions. First, the overlapping goals and functions of these 
institutions create the need for accountability systems that can 
address different and sometimes competing goals (Smith, 1991). 

A related challenge is that of multiple student bodies. Postsecondary 
institutions and their vocational programs serve several groups of 
students simultaneously. These include traditional, full-time 
students looking to transfer to four-year institutions; young adults 
seeking higher-paying employment options; middle-aged adults 
seeking retraining and career change; and retired adults investing in 
education for the sake of personal fulfillment Each of these groups 
seeks different outcomes and requires different outcome measures 
(Ewell, 1989). Consequently, designing outcomes assessment 
around highly structured curriculum paths is extremely difficult to 
accomplish. This is because, according to Adelman (1992), 
students who come to colleges with very different needs do not take 
the curriculum the way it was designed They simply do not 
participate in a lock-step fashion the way students might in 
secondary education. 

Furthermore, a large percentage of the students participating in 
vocational education are adults who are voluntarily participating in 
the educational experiences. According to Ewell (1992), community 
college "students don't stand for assessment." He advocates 
assessment systems that are infused into the curriculum so that 
students are not intimidated by formal testing procedures. 
Alternative assessment approaches that enable students to 
demonstrate skills and knowledge in natural settings and at 
convenient times and places can help to accomplish this goal. 

How can these and other assessment related challenges be 
addressed? Alfred and Kreider (1991), Simmons (1991), and 
(Sims, 1992) make the following recommendations for developing 
outcomes assessment systems: 
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Recommendations for Creating 
an Outcomes Assessment System 



• Develop a clear, focused, and realistic mission statement that 
can provide the foundation for establishing goals as the basis 
for outcomes assessment and institutional effectiveness 
processes. 

Ensure that issues of cultural diversity and pluralism are 
adequately addressed for all faculty, staff members, students, 
governing boards, curricula, and services throughout the 
institution. 

Obtain commitment from the top of the institution to support the 
outcomes assessment processes with resources and make 
faculty the driving force behind development of assessment 
processes to ensure that useful information is created for all 
levels. 

Become a customer-driven organization by defining internal and 
external customer groups and ensuring that meeting their needs 
and expectations is the institution's top priority, 

• Audit existing information collected about the institution's 
internal and external environments to determine what is known 
about how processes arc functioning; reinforce and coordinate 
these essential data collection activities. 

Create longitudinal databases using qualitative and quantitative, 
institution-specific data on student performance, course taking 
patterns, and short- and long-term results from student 
participation. 

Conduct targeted evaluations to determine how individual 
programs are working and empower faculty, staff, and students 
throughout the institution to carryout these types of evaluations 
to make continuous improvements in programs. 

Develop better information about the infrastructure of the 
institution for planning and program improvement and ensure 
that this information is widely disseminated. 

Develop mechanisms for feeding assessment results back into 
the system for use in stimulating innovation and improving 
policies, programs, services, and student outcomes. 
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All of these recommendations can be accomplished through the use 
of specific outcomes that axe measured throughout the institution, 
especially at the program and classroom levels. To address the 
challenges posed to practitioners in this chapter, information 
presented in the remainder of this book provides perspectives on 
approaches to assessment of outcomes at die institutional, program, 
and student levels. 
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Chapter 2 

Total Quality Management 

Debra D. Bragg 



Overview Total quality management (TQM) is being adopted by American 

business and industry to improve quality and increase economic 
competitiveness. Educators have only recently begun to examine 
TQM and to consider its implications for educational reform and 
accountability* Little is known about how ideas regarding customer 
satisfaction, process management, and employee involvement apply 
to education, especially outcomes assessment. Are these ideas 
compatible with education? Can they improve quality in 
postsecondary education? Parallels between TQM and outcomes 
assessment and other related issues are explored in this chapter. 

In this chapter This chapter discusses the following topics: 



Topic See Page 

Guiding Principles of Total Quality Management (TQM) 16 

Five Goals of TQM 17 

How TQM Relates to Outcomes Assessment 2 1 

Implementation of TQM 23 

Implementation Issues 25 

TQM at Fox Valley Technical College 26 
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Guiding Principles of Total Quality 
Management (TQM) 



TQM is a 
comprehensive 
and process- 
oriented 
approach to 
outcomes 
assessment 



Traditionally, higher education institutions have defined quality by 
their "faculty, facilities, support, curriculum, and student attributes" 
(Litten & Hall, 1989, p. 304), particularly in relation to their peer 
institutions. Consequently, the focus of measurement has largely 
been on those characteristics of higher education that mattered most 
to faculty and administrators within the system. Attention paid to 
external audiences has grown, however, as government, private 
funders, and taxpayers have increasingly demanded more 
accountability. 

How is quality assessed in postsecondary vocational education 
settings? Quality can be assessed, as Harmon points out later in 
Chapter 4 of this document in his discussion of the value-added 
concept, by measuring changes in students between the time they 
enter and leave a vocational education program; or as McCaslin 
points out in Chapter 6 of this document, by measuring results of 
student participation evidenced by economic, educational, or psych- 
social outcomes; or as Mabry discusses in Chapter 7, by measuring 
how students actually perform specific tasks acquired in vocational 
education. In all cases, these approaches to assessment focus on 
end results; they determine whether desired outcomes have been 
achieved for programs and students. 

Without a doubt, total quality management (TQM) is concerned with 
end results. However, they are not the only consideration. TQM 
views outcomes assessment as having a place in determining quality; 
however, its view is much more comprehensive and process- 
oriented (Bryce, 1991a). In the TQM approach, outcomes 
assessment is conducted for the purpose of refining, managing, and 
measuring work processes so that customer needs can be met 
Some define quality as "the ability of a product or service to meet 
consistently the expectations of customers" (Ackerman, Coleman, 
Leger, & MacDorman, 1988, p. 5). Therefore, meeting customer 
needs and ensuring student outcomes through execution of well- 
managed processes becomes a primary concern throughout all levels 
of an institution. 



The concept of continuous improvement is also central to 
understanding TQM, as is typified by Procter & Gamble's definition 
of quality, "the unyielding and continually improving effort by 
everyone in an organization to understand, meet, and exceed the 
expectations of customers" (Procter & Gamble, 1989, p. 1). TQM 
is characterized by its rigorous process for making small, 
incremental improvements through the use of group process, data 
analysis, and statistical tools (Bryce, 1991a). 
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TQM may 
assist 
employers 
and 

educators to 
reach higher 
standards 



TQM is a fairly recent phenomenon in the United States, especially 
in education* General principles that drive TQM evolved primarily 
from die philosophies of two Americans who worked extensively 
with Japan after World War TL W. Edwards Deming and Joseph 
Juran are credited with inspiring post-WWH Japan to adopt the 
quality methodology and eventually turning that country into a world 
economic competitor. Only in recent years have American business 
and industry begun to seriously consider the recommendations of 
Deming and Juran* Even fewer American educators are aware of 
TQM or considering its implications for educational restructuring 
and accountability (Brandt, 1992), 

Why is the time right for educators to consider quality management 
approaches? Recent attention focused on the role of employers and 
educational systems in economic competitiveness has heightened 
comparisons between reform in both these arenas (Kirby & Bragg, 
1992). Representative of the calls for reform, the Commission on 
the Skills of the American Workforce (1990) has recommended 
employers adopt high-performance workplaces and the Secretary's 
Commission on Achieving Necessary Skills (SCANS) has 
encouraged educators to ensure "world-class standards of 
curriculum content and student performance" (SCANS, 1992, p. 
xix). TQM may provide an approach to assisting both employers 
and educators to reach higher standards* 



Five Goals of TQM 



What are the fundamental goals of TQM? They focus on mobilizing 
everyone in an institution to 



manage work processes, 

exceed customer expectations, 

ensure a systematic approach, 

measure for continuous improvement, and 

become involved in the entire process. 



Managing Work Processes 

A primary goal of TQM is to manage the flow of all work processes 
so that customer expectations are continually met or exceeded (Sherr 
&Lozier, 1991). A process can be viewed as "a set of interrelated 
work activities that are characterized by a set of specific inputs and 
value-added tasks that produce a set of specific outputs" (Ackerman, 
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Coleman, Legcr, & MacDorman 1988, p. 5). All work can be seen 
as fitting into various types of processes and all employees can be 
viewed as managers of these processes. Applied to education, 
processes of postsecondary institutions such as admissions, 
registration, counseling, instruction, curriculum development, and 
job placement are the focus of process improvement 



Work processes 
contribute to or 
hinder student 
expectations 
and outcomes 



TQM focuses on how work processes contribute to or hinder the 
realization of customer expectations (Ackerman et al., 1988). A 
fully-functioning TQM approach yields outputs (e.g., products and 
services) that are responsive to ever-changing customer needs. 
According to a TQM philosophy, a college's institutional 
effectiveness can be determined by measuring the effectiveness and 
efficiency of these processes. All work processes, viewed as 
operating within a system, contribute to meeting student needs and 
expectations, thereby producing desirable outcomes. The goal of a 
top-notch TQM. approach is to ensure that these educational 
processes are continually refined and refocused on meeting student 
needs and expectations. 



Exceeding Customer Expectations 

A common theme of TQM is that customer needs drive quality 
improvement. Who ate the customers of postsecondary vocational 
education? First and foremost, they are students. A focus on 
knowing student needs and expectations and continually trying to 
meet or exceed those needs is key to improving quality. Rhodes 
(1992) is critical of education's perspective toward students. He 
explains that "schools have been 'student-centered* in the same way 
that the work of a basketball team might be called 'hoop-centered.'" 
(p. 77). He describes educators as well intentioned but isolated and 
somewhat selfish in their attempts to win the game. Each educator, 
he explains, is shooting at the basket when he or she gets the ball. 
Even though each makes some baskets, the team usually loses. He 
concludes by saying, 

What does that metaphor have to do with paradigms for 
education? Keeping the student foremost in our thoughts 
has little to do with shifting our sense of the system. We 
still are looking at the student. The total quality view allows 
us to see with the student's eye view — to understand what 
the school and world around it looks like to children 
growing up today (Rhodes, 1992, p. 77). 

The focus on customers is pervasive in Japanese business thoughts 
and actions. Japan was first in defining customers as both internal 
and external (Bryce, 1991b). Modern marketing practice in the 
U.S. also emphasizes the customer. Boone and Kurtz (1985) define 
marketing as "a company-wide consumer orientation with the 
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^ A focus on 
customers is 
pervasive in 
Japanese 
business and in 
modern U.S. 
marketing 
practices 

By taking this perspective, customers of all processes can be viewed 
as within or outside of the system. Therefore, there are many types 
of customers of postsecondary education besides students. 
Seymour (1992) describes three sets of individuals or groups that 
play a role in quality as we've traditionally viewed it for higher 
education. They are (1) regional and discipline-based accrediting 
agencies, (2) college administrators, and (3) college faculty. In 
addition, other customers who typically play an internal or external 
role depending upon their level of involvement with an institution 
are community organizations, government agencies, foundations, 
private agencies, and taxpayers. 



objective of achieving long-term success. The key words are 
company-wide consumer orientation. All facets of the organization 
must be involved with assessing, and then satisfying, customer 
wants and needs" (p. 13). The process of converting needs into 
wants is seen as central to the purpose of modem marketing. TQM 
clearly adopts this perspective since it advocates that every employee 
work to understand, meet, and exceed customer needs and 
expectations. 



Deming and 
Juran view an 
entire system 
in need of 
continual 
improvement 



Ensuring a Systematic Approach 

The notion of a system is prominent in the writings of Deming 
(1986), Juran (1989), and others involved with developing TQM 
approaches. They view the entire system as in need of continual 
attention if improvements are to occur for any one of its parts. 
Systems theory focuses on processes — their inputs, outputs, and 
feedback — and the emphasis on the interdependency of these parts 
(Stimson, 1989). Checkland (1972) defined systems theory as the 
"complex grouping of human beings and machines for which there 
is an overall objective" (p. 91). Applied to education, systems 
involve the interrelated workings among people, processes, 
products, and other phenomenon for the purpose of achieving goals 
and producing outcomes. 

According to Deming (1986), an organization's central activity 
(e.g., production in manufacturing or instruction in community 
colleges) must be viewed from a systems perspective. If any one 
part of the system breaks down, quality is jeopardized. He explains 
that constant improvement of quality must be pervasive throughout 
an entire organization from the time materials enter until the time 
they reach the customer. Products and services must constantly be 
redesigned for the future. "It is this customer-driven, team-fueled, 
and even-keel approach to business that forms the basis for the 
constancy of purpose that Dr. Deming is urging" (Scherkenbach, 
1986, p. 10). 



Deming believes it is the responsibility of the institution to design a 
system that anticipates customer needs on an on-going basis. Yet, 
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he also contends that customers are largely unaware of what will 
constitute their future needs* Therefore, an important responsibility 
of organizations is to use research and development to create 
innovations that can address and even produce future customer 
needs* Innovation occurs through change in products, processes, 
and policies* Deming explains that "die moral is that it is necessary 
to innovate, to predict needs of the customer, to give him [sic] 
more" (Yates, 1992, p* 21)* The importance of Deming's 
perspective for education is to create systems and processes that 
guide instructional innovations and to continuously measure how 
they relate to student outcomes* 



Measuring for Continuous Improvement 



The threshold 
for quality is 
continually 
raised as 
customers 
receive better 
products from 
the institution 



A fundamental concept of TQM is that outputs are never good 
enough; the threshold for quality is continually raised as customers 
receive better services from the institution* On-going measurement 
is required to determine how a process is working and to make 
improvements when problems occur* This happens through use of 
measurement indicators that are extremely sensitive to whatever 
produces or impedes high quality end results (e.g., student 
outcomes in education), "The central concept in applying statistics 
to measurements is that every measurement is made in the context of 
a measurement system. A measurement system includes not only 
the object to be measured and the measurement equipment, but also 
the procedure and context of measurement (Coleman & Stein, 1990, 
p* 35)* Deming believes that by understanding how variation occurs 
in the context of processes, products, or systems, it is possible to 
make improvements* The theory of variation is the foundation of 
Deming's management philosophy; his desire to reduce variation in 
the way processes — not people — function within the organization is 
a part of each of his fourteen points (Joiner & Gaudard, 1990)* His 
purpose is not to reduce variation to the point of stagnation but, 
rather, to ensure a minimum and maximum level of quality in all 
products and services* 

In the application of measurement to service organizations, such as 
educational institutions, the ability to identify and manage quality 
improvement through the use of group processes, tools, and 
problem solving plays a critical role in measuring for continuous 
improvement In discussing the implications of TQM for outcomes 
assessment, Seymour and Chaffee (1992, p* 26) explain that "data 
are part of a diagnostic process, not an accountability weapon*" 
Therefore, the systematic use of measurement through both 
quantitative and qualitative methods can inform further measurement 
for determining outcomes for students. 
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Involving People in the Entire Process 



A quality 
infrastructure 
touches the 
entire 

organization 
and ensures 
that work 
processes are 
in line with 
instructional 
goals 



The fifth and most fundamental concept of TQM is that quality 
improvement must be accomplished through the knowledge and 
dedication of people employed within the institution* While ideas 
for innovation can come from internal or external customers, the 
actual effort of carrying out quality improvement and process 
management activities is accomplished by those employed within an 
organization. Theories regarding participation and employee 
involvement are based on the belief that "employees will take more 
pride and interest if they are allowed to make meaningful 
contributions to their work and influence decisions made about it" 
(Aubrey & Felkins, 1988, p. 1). 

It is through collective decision making, where each person's 
expertise is recognized and valued, that organizations are enabled to 
make improvements. Typically, responsibilities are differentiated 
depending upon where an individual is employed in the institution 
but not to the extreme that roles are differentiated as they are in 
bureaucratic organizations. Rather, senior administrators may be 
involved in policy making, and mid-level administrators may also 
influence policy as well as set goals and specify performance 
indicators. Front-line workers may be involved in determining 
performance indicators, monitor how these relate to customer needs 
and expectations, and feed this information back into the system for 
others to use. 

Altogether, an institution's employees create a quality infrastructure 
that touches the entire organization and ensures that all work 
processes are operating in line with the institution's goals for 
continuous quality improvement. Employee involvement at this 
level requires extensive training so that everyone in the institution 
can acquire skills and knowledge regarding the quality process. 
Employees are trained in how to use group processes, problem 
solving, quality tools (e.g., brainstorming, cause and effect 
analysis, pareto analysis), and research and statistical analysis 
techniques; and they are provided with organizational, individual, 
and team leadership skills. The idea of involving teams in the 
workplace evolves from the need to facilitate employee involvement 
to capture expertise for making quality improvements. 



How TQM Relates to Outcomes Assessment 



While difficult to generalize, a comparison of TQM to educational 
outcomes assessment approaches such as performance assessment 
or value-added assessment reveals that TQM is more comprehensive 
in its approach and more highly focused on evaluation for the sake 
of improvement than for the sake of accountability. Accountability 
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TQM is highly 
focused on 
improvement; 
yet, it creates 
information 
needed for 
accountability 



to customers is a necessary element of the TQM approach, however, 
it is not all encompassing. Without a doubt, improvement is the key 
concept In the view of Seymour and Chaffee (1992), 

Assessment shifts the focus on quality from inputs to outputs, 
seeking to measure the extent to which educational programs 
produce intended results: adequately educated graduates. This 
narrow focus onstudent outcomes, popular among states and 
boards, may satisfy the demand for accountability. But 
assessment, if done as part of a larger quality improvement 
effort, also can aid institutional effectiveness." (p. 26). 



TQM shifts assessment to viewing outcomes as evidence of the 
effectiveness of the whole system. To understand the parts requires 
an understanding of the whole. Therefore, the focus of assessment 
must be on indicators of performance of the system's key work 
processes and how each contributes to end results. By looking at the 
whole, including measures student outcomes, data can be fed back 
into the system to make improvements, as ""?11 as to address 
accountability issues. In a fully-functioning TQM system, the entire 
institution is dedicated to this effort 



Juran's quality* 
planning road 
map applies 
measurement 
to all levels of 
the system 



A quality-planning road map designed by Juran (1989) illustrates this 
approach (Figure 1). Measurement is applied throughout the system, 
not just to customer needs or product features. Measures are also 
applied to process development and data generated are fed back into 
the system at all levels. 



Editing Product and Process 
►I Identify Custom en 



list of Customers 

+ 

*—+-\ Discover Customers* Needs | 

List of Customers' Needs On their Language) 

—1= * 



Translate 



J 



Customers' Needs (in our Language) 

* 

►I Develop Product j 


Product Features 

♦ 

Develop Process 1 

r 



Process Features (process ready to produce) 
— a» | Transfer to Operations } 



Figure 1. Juran's quality-planning road map (Juran, 1989) 
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Developing a 
quality 
measurement 
system is 
pivotal to 
understanding 
the workings 
of the entire 
system 



A measurement approach that focuses on the entire system was 
developed by Rummler and Brache (1990). Their conceptualization 
involves developing a quality measurement system to 

• identify significant outcomes of the institution, its processes 
(e.g., counseling and instruction), and its jobs (e.g., instructor, 
dean, and president); 

• identify critical dimensions of performance (e.g., accuracy, 
relevance, timeliness, and cost) for all significant outcomes; 

• develop indicators for each critical dimension of performance; 
and 

• develop measures and standards for each indicator. 

Note that the system does not focus on end results or student 
outcomes alone* It includes these measures, of course, as pivotal to 
understanding the workings of the entire system. However, the 
approach of Rummler and Brach goes well beyond end results to 
measuring outcomes of jobs, processes, and the institution as a 
whole. 



Implementation of TQM 



Juran's eight 
prerequisites 
provide 
guidance for 
implementation 
of TQM 



Juran's eight prerequisites for strategic quality management (SQM) 
provide a model for implementation of quality improvement and 
process management approaches. The eight prerequisites Juran 
(1990) employs are 

1 . Provide leadership from executive staff 

2 . Establish the quality vision and policies 

3 . Establish broad quality goals 

4 . Deploy the quality goals to all levels of the organization 

5 . Provide the needed resources, including training 

6 . Establish measurements 

7 . Review performance regularly 

8. Revise the reward system to give adequate priority to quality 
improvement 
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Provide 
leadership for 
TQM throughout 
the institution 



Deployment of 
quality goals 
moves TQM 
into the hands 
of people who 
can implement 
it 



Kouzes and Posncr (1990) have pointed out that it is the primary 
contribution of leaders to recognize and support good ideas of front- 
line workers and "challenge the system in order to get new products, 
processes, and services adopted" (p. 8)- It is the responsibility 
of leaders at all levels of the institution to become involved in 
implementation of TQM and to formulate a vision and communicate 
openly and enthusiastically about it It is the job of senior leaders to 
support TQM with fiscal, material, and human resources to ensure 
that it is implemented fully* 

A vision of the institution's beliefs about quality and policies to 
support them are essential for unifying people and creating 
enthusiasm and a sense of importance* The development of a 
shared vision and goals for TQM requires personal commitment by 
every employee to thinking about how one feels about ideas and 
repeatedly engaging in conversation to express those ideas until 
consensus is reached Once a vision and policies for TQM are 
adopted, it is critical for an organization to examine how they relate 
to other institutional policies and practices and to ensure congruence 
among these policies. 

The next step is to establish and prioritize goals for TQM. These 
goals may start as a "wish list" generated by all employees of an 
institution. However, this list must be paired down and prioritized 
to create a master list of quality goals for the institution. To be 
effective, these goals should be stated in specific, quantifiable terms. 

The idea behind deployment of quality goals is to move TQM into 
the realm of responsibility of people who can implement it in their 
day-to-day work processes. Through quality improvement teams, 
process management teams, and other similar problem-solving 
groups, employees can become actively engaged in TQM. Of 
course, participation by all employees in TQM requires a clearly 
specified organizational structure for the system as well as open 
lines of communication throughout the institution. 

TQM cannot be implemented effectively without a commitment of 
resources in people, time, materials, and sometimes even facilities. 
Training in the use of teamwork, problem solving, and measurement 
are crucial to making TQM work effectively in an institution. As 
employees become increasingly sophisticated in their use of TQM 
processes, opportunities for further knowledge and skill are 
required. Therefore, while the opportunity exists to improve 
efficiency and reduce costs in some areas of the institution, other 
costs are sure to rise, making a long-term commitment necessary to 
making TQM of value in an educational institution (Keller, 1992). 

It is important to use measurement to ensure that TQM is meeting its 
intended goals. A ten-step strategy for carrying out measurement of 
the effectiveness of TQM is described by Kirby and Bragg (1992). 
This strategy involves documenting the goals of TQM, building on 
existing measures, using qualitative and quantitative data collection 
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methods, and continually refining measures to reflect higher 
standards of performance. 



On-going 
review and 
revision of 
reward systems 
is crucial to 
TQM 



On-going review of performance is critical to establishing an 
effective TQM approach. Reviews should focus on determining the 
degree to which quality goals and outcomes are being met and 
improved upon. When problems arise, feedback should be 
provided to individuals who can modify the system to ensure quality 
improvement 

Many educational institutions employ reward and incentive systems 
that contradict the TQM philosophy. These may include tenuring of 
faculty, merit pay for faculty and administrators, and competitive 
grading of students. New reward systems must be developed to 
provide incentives that motivate employees to manage work 
processes consistently with the goals of TQM and to ensure that 
high standards are obtainable by all. 



Implementation Issues 



TQM is being 
implemented to 
improve 
educational 
systems and to 
meet 

accountability 
demands 



In a limited capacity ,TQM is increasingly viewed as a potential 
approach to improving educational systems and meeting account- 
ability demands (Bemowski, 1991; Bonstingl, 1992; Brandt, 
1992; DeCosmo, Parker, & Heverly, 1991; Keller, 1992; Litten & 
Hall, 1989; Miller, 1991; Seymour, 1992; Seymour & Chaffee, 
1992; Spanbauer, 1992). In 1991, Axland identified fourteen 
community and technical colleges involved with TQM of which 
sixty-four percent were implementing TQM within their own 
administrative units. Slightly less than half of the institutions were 
offering TQM courses in management or technology curriculum 
areas. Even fewer were developing curriculum to prepare 
individuals for occupations created by the quality movement (e.g., 
quality consultants and statistical process control (SPC) 
technicians). Consequently, whereas TQM appears to have a 
foothold in a few institutions, it seems it has a long way to go before 
being recognized as common practice in the majority. 

As educational institutions move forward with implementation of 
TQM, issues are being raised. Some issues pertinent to TQM 
implementation in postsecondary education are suggested here, 
although it is premature to offer solutions. The purpose of raising 
the questions is to stimulate discussion about TQM in order to assist 
institutions to consider the realities of TQM implementation. 
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Implementation Issues 



How can a postsecondary institution make TQM fit within 
existing management and evaluation systems without losing the 
integrity of traditional approaches? 

• Is TQM only applicable to administrative and support 
processes? What does it offer teaching and learning processes? 

• How can an institution handle the extensive requirements for 
education and training about TQM of its own staff? 

• Where will institutions find the time to involve employees in 
quality improvement activities? 

• When the needs and expectations of different customer groups 
are in conflict, which groups get priority? 

• How can an institution move to a focus on processes when 
traditional accountability demands force it to differentiate 
programs (e.g., evaluating vocational or transfer programs 
rather than processes associated with both such as guidance or 
registration)? 

• Can efforts to control process variation impede creativity and 
responsiveness and an institution's ability to address 
increasingly diverse student needs? 

• How can an institution avoid assessing what's easy to measure? 

• With increasingly limited resources, who will pay for TQM? 



TQM at Fox Valley Technical College 



Of the several postsecondary educational institutions that have 
implemented TQM, Fox Valley Technical College (FVTC) in 
Appleton, Wisconsin, is probably one of the best known. Fox 
Valley embarked on a quality journey in 1985 when a local business 
representative suggested the college provide training in quality 
management, specifically in zero defects management. After 
researching the fundamental concepts of quality and investigating 
how local business and industry was getting involved, the college's 
president, Stanley Spanbauer, and administrative staff set out to 
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understand what the quality methodology would entail for FVTC 
(Spanbauer, 1992). 



A sixteen-step 
quality 
improvement 
model 

incorporates 
the best of 
several TQM 
experts, 
according to 
FVTC staff 



During 1985, the College was awarded a small grant from 
Wisconsin's office of Vocational/Technical Education to design an 
introductory quality and productivity curriculum for potential use 
across the state. In addition, a college-level task force along with a 
study by Jo Hillman, FVTC executive assistant, recommended the 
college deliver curriculum offerings in quality and productivity for 
local business and industry. In 1986, Spanbauer approached the 
college's board for support of a three-year, $60,000 proposal to 
implement a quality improvement process at FVTC. Spanbauer has 
recently written about this experience, "To my surprise, there were 
only a few questions after my presentation, and the board endorsed 
the initiative. Our journey in quality had begun!" (Spanbauer, 
1992). 

Since that time, FVTC has developed and refined its quality 
improvement process into a sixteen-step activity model which its 
staff thinks incorporates the best of several TQM experts. The 
sixteen steps follow: 

1. Demonstrate management commitment 

2. Establish a total quality leadership council 

3. Determine the cost of quality 

4. Provide education and training 

5. Identify roles and establish performance requirements 

6. Implement a quality communication system 

7. Measure and set goals 

8. Identify and eliminate problems 

9. Research and develop new initiatives 

10. Create a structure for employee initiatives 

11. Establish accountability 

12. Launch a customer revolution 

13. Recognize, reward, and celebrate 

14. Conduct quality audits 

15. Link to the community 
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Four elements 
of TQM are of 
importance to 
this discussion 
about FVTCs 
approach 



16. Strive for continuous improvement (Spanbauer, 1992) 

Four elements of the quality improvement process are of particular 
importance to this discussion. First, dealing head on with the issue 
of education and training, FVTC continues a strong commitment to 
educating its own staff in quality principles and methods. During 
Phase I of TQM implementation, every employee received twenty 
hours of quality awareness training. Now, during Phase EL, all 
employees receive thirty-six hours of training, especially in the areas 
of process improvement and management and statistical process 
control. This training is provided by the college's own Quality 
Academy. 

Phase II training is offered to groups of personnel who have similar 
jobs and responsibilities. All groups receive training in team 
building, process management, problem solving, and customer 
service; the college mission, purpose and directions; and quality 
review concepts. In addition, each personnel level has major 
emphasis areas: managers receive intensive training in process 
improvement and statistical process control; faculty are instructed in 
teaching effectiveness, outcomes assessment, and classroom 
management; and service and support staff get instruction in 
customer service. 

The second element involves the structure of TQM at FVTC, 
specifically the use of a total quality leadership council. Individuals 
are elected to the council for one-, two-, or three-year terms. It is 
comprised of three administrative staff, three support staff, and three 
faculty in addition to President Spanbauer. Established fairly 
recently, the council helps to monitor the quality improvement 
process and develop a structure for its continued operation. The 
Council's current purpose is to promote the quality initiative and 
ensure its effectiveness for the entire institution. 

Third, an important focus of the FVTCs quality initiative is to 
improve it's processes, products, and services. Quality and process 
improvement are addressed on several levels. Directly accessible to 
students — FVTCs primary customers — but available to all of 
FVTCs internal and external customer groups are comment cards 
that provide a means of reporting going right or wrong in any aspect 
of campus life. Since these cards are strategically located on campus 
and picked up two or three times per week, the College has a direct 
and anonymous mechanism for monitoring and managing customer 
satisfaction. Focus groups are also used to collect data on student 
needs and expectations. 

In addition, problem-solving teams are formed within units and 
across the campus to enable employees to take corrective action 
when problems are detected. Teams directly confront problems that 
can potentially disrupt work processes and ultimately jeopardize 
students' opportunities to obtain positive outcomes. Whereas 
serious problems can be escalated to higher levels, FVTC finds that 
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eighty percent of its problems can be resolved by these front-line 
problem-solving teams (Bemowslri, 1991). FVTC identifies eight 
quality elements as the basis for its quality initiative: 

• human resources 

♦ curriculum and instruction 



goal setting 
use of technology 
marketing 
customer service 



management 
trustee/board 



For each of these quality elements, the college identifies conforming 
equirements, measurement strategies, and costs of 
nonconformance. These elements are used in annual self analysis 
by staff and faculty to select problems to tackle. Monitoring these 
elements is important to ensuring continuous operation of the 
college's quality initiative. Teams of business and industry people 
conduct audits to determine how well the college is meeting 
standards set for each element. 



Measurement 
is critical to 
making 
improvements 
and 

demonstrating 
accountability 



FVTC is committed to using measurement for the purposes of 
making improvements and demonstrating accountability. This is 
done by merging quality improvement and process management 
efforts with the colleges' traditional institutional research activities. 
Measurement strategies employed at FVTC on a regular basis are 
presented below. Through this extensive measurement process, 
FVTC strives to manage and improve quality throughout the 
institution. To attest to this commitment, the college offers 
educational guarantees to its graduates and local employers. The 
college offers retraining or refunds if these guarantees are not met 



FVTC Measurement Strategies 



• Indicators of district health (e.g., enrollments, withdrawals, and 
new contracts) are reported on a monthly basis. 
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♦ Graduate follow-up, instructional audits, service audits, needs 
assessment, information management studies, and financial 
studies are conducted on an annual basis. 

♦ Organizational climate, student satisfaction, and employer 
surveys are administered on a biennial basis. 

♦ Community perception surveys, competitive market analysis, 
labor/business market trends analysis, and other strategies are 
used on a three- to five-year basis. 



When interviewed about visible changes that have occurred at FVTC 
since TQM was first employed in 1985, Callie Zilinsky, Director of 
FVTCs Quality Academy, identified several. Zilinsky (1992) 
explained that die college had undergone a cultural transformation; it 
had moved from a campus divided by management and labor 
disputes to one characterized by a real team perspective. She 
described planning and management efforts that routinely involved 
committees with representatives from throughout the college. 

Second, Zilinsky felt there had been skill building among staff and 
described FVTC as a learning organization. Consequently, 
employees were more receptive to learning new things, especially in 
how to carry out evaluative activities to make quality improvements. 
She explained that, over the years, staff were showing steady but 
gradual improvements in comfort with and use of measurement. 
Finally, she felt that students were being treated differently because 
of TQM. Through use of comment cards and other problem-solving 
teams, students were feeling some of the benefits of FVTCs quality 
initiative. 

What are the challenges that FVTC continues to face in its quality 
journey? Zilinsky spoke of two. First, cost constraints are a reality 
in higher education, she said. Expressing fears of reduced 
educational funding, she said, "Cost containment scares me to 
death." Her concerns were that institutions would use accountability 
measures currently mandated by funding agencies, resulting in a 
limited set of measures driving institutional decision making in lean 
fiscal years. In such times, people and departments are likely to be 
pitted against one another, rather than working toward a goal of 
improved quality for the entire institution, according to Zilinsky. 
She also pointed out there is a tendency for measures to shift from 
year to year as political emphases change, once again resulting in an 
uncertain focus for an institution's quality improvement efforts. 
Particularly strong and committed leadership is needed to ensure that 
a TQM effort stays on track during difficult financial times, 
according to Zilinsky. 

The other challenge identified by Zilinsky was one described by 
some others attempting to apply TQM in education. That is, "What 
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Challenges 
remain at FVTC 
for full 

implementation 
of TQM 



docs TQM mean for improving teaching and learning?" She sees 
these processes as critical to the quality of a college like FVTC and 
showed concern for limiting TQM to only administrative processes. 
While she admitted to not having all the answers, she said that she 
believes there are many, yet untapped faculty resources that will 
make the connection between TQM and teaching and learning* 

Through increased use of group processes involving faculty and 
students, there will be more exploration of how TQM applies to 
teaching and learning. The possibilities for improved quality in 
teaching and learning through application of TQM are largely 
unknown at this time. However, it seems apparent that improving 
these critical processes can potentially have the most direct impact on 
student outcomes. Advancements in assessment approaches to 
measure improvements in teaching and learning represent the 
promise of TQM for education. 
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Chapter 3 

Assessing Student Success 

C. Michael Harmon 



O verview One challenge of outcomes assessment is to create an environment in 

which assessment contributes to the development and maintenance 
of institutional effectiveness and student success. One way to 
accomplish this is to develop an institutional climate in which 
student success is seen as an important goal and then to gear the 
function of all elements of the institution to meeting this goal. 
Outcomes assessment is an important part of this effort. Without 
ownership by all members of a college, outcomes assessment 
policies will not be fully implemented This chapter looks at how 
one college is implementing this process to ensure institutional and 
instructional quality. 

In this chapter This chapter presents the following topics: 



Topic See Page 



Institutional Effectiveness = Student Success 36 

Santa Fe Community College 36 

Student Outcomes Studies 38 

SFCCs Instructional Program Review Process 39 

Institutional Effectiveness and Student Success 40 

Safety Checks 41 

Fostering Teaching Effectiveness 42 

SFCCs Outcomes Committee 43 

Uses of Outcomes Study Reports 44 

Implementation Issues 45 

Exporting the Process 48 
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Santa Fe (New Mexico) Community College in the preparation of this chapter. 
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Institutional Effectiveness = Student 
Success 



An important 
measure of 
institutional 
effectiveness 
student goal 
attainment 



is 



Student success 
requires 
support and 
reinforcement 
from all college 
employees 



Increasingly, an important measure of institutional effectiveness at 
community colleges is whether students are able to successfully 
meet the goals that inspire them to attend the institution. For some 
faculty and administrators, emphasizing student success requires 
making a fundamental change in operating paradigms. 
Traditionally, many postsecondary educators have operated on the 
premise that students have the right to take any class and to pass or 
fail that class. Some have even felt that a high failure rate is 
necessary to achieve high standards (Dressel, 1961). 

The student success paradigm operates from a vastly different 
perspective. Borrowing from the principles of Total Quality 
Management (TQM) and other quality initiatives, it sees the student 
as a customer and the college as a provider of services. 
Consequently, student success focuses on identifying student goals 
and devising a course of study and a program of support services 
that enables students to meet their goals and then monitoring 
students' progress toward meeting those goals. Exit interviews or 
other means of assessment may be used to determine whether 
individual student's goals were met. 

The climate needed to support student success does not develop 
without effort and student success cannot achieve its full potential 
without broad-based institutional consensus. The process is time 
consuming and requires constant support and reinforcement 
beginning at the highest levels of administretion and extending 
throughout the entire institution. Faculty, administrators, and 
clerical personnel all have direct contact with students and other 
stakeholders and influence their perception of the campus. So, too, 
do cafeteria workers, maintenance and custodial workers, security 
officers, and others. Successful implementation of the student 
success paradigm requires the involvement of all college employees. 



Santa Fe Community College 



One college that has implemented the student success model is Santa 
Fe Community College (SFCC) in Santa Fe, New Mexico. 
Established in 1983, SFCC set out to implement this model even 
before it opened its doors. This, some practitioners have said, made 
it unusually easy for college administrators to put into place any type 
of program they wanted; this institution had no inertia to overcome. 
This might well be true if all of their employees were novices when 
they were hired. They were not. And while efforts were (and 
continue to be) made to hire staff that would be receptive to this 
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approach, some veterans came into the program with serious 
philosophical reservations about its efficacy and propriety. 

To the extent that the inertia to be overcome rests not in the 
institution but in the paradigms used by its employees, SFCC began 
from the same starting point as any other institution. Indeed, other 
older institutions — for example, Jacksonville Community College in 
Jacksonville, Horidan-are developing similar programs. 



Background on Santa Fe Community College 

Santa Fe Community College is an open admission, comprehensive 
public community college and is accredited by the North Central 
Association's Commission on Institutions of Higher E ducati on. Its 
fall semester 1991 enrollment included 3,552 (1,131 FTE) credit 
and 2,628 non-credit students. The college also served 2,200 adult 
basic education students (Santa Fe Community College, 1991b). 
Nearly sixty-five percent of the credit students were female fifty-two 
percent were white, forty-two percent Hispanic, and three percent 
Native American, Part-time (less than twelve credit hours) 
nondegree-seeking students accounted for seventy-three percent of 
die student body, followed by part-time degree seekers (14%), full- 
time degree seekers (7%) and full-time nondegree seekers (5%). 
Academic courses accounted for fifty-five percent of the 16,965 
credit hours taken by credit students. Vocational courses accounted 
for twenty-six percent of the total, with developmental studies 
providing the remainder (New Mexico Commission on Higher 
Education, 1991). 



Outcomes Assessment at SFCC 



Outcomes 

connect 

student 

achievement 

to college 

mission 



The SFCC approach to outcomes assessment connects student 
achievement directly to both the college's mission and specified 
learning objectives for each discipline, course, and program. 
Achievement is measured while the student is enrolled at the college 
and after the student leaves the college. This practice allows the 
college to assess the extent to which it actually teaches what it says it 
teaches and the extent to which students actually learn what is taught 
and what they need to know to meet the goals they have set for 
themselves. 



For example, course success and correlational studies of student 
achievement in math and English courses are used to ensure that 
specified competencies are, in reality, being acquired while the 
student is in attendance. These studies also define the degree to 
which learning objectives are achieved by students individually and 
collectively in their classes. The degree of success experienced by 
students as measured by criterion referenced exams becomes a 
quality benchmark for both the college and the individual student 
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and allows the college to be confident of the quality of its course 
offerings and curriculum sequence. 

The college also measures student success after enrollment 
concludes. Studies at SFCC show that students who transfer to a 
four-year college tend to maintain or improve the grade point 
average they earned at the community college. Further, SFCC 
transfer students tend to perform as well as or better than the 
receiving school's native students, as measured by grade point 
average (Santa Fe Community College, 1988b). 

Studies also show that vocational program completers do well on 
licensure exams (Santa Fe Community College, 1988a; 1991a) and 
that graduates of programs leading to licensure have a ninety-eight 
percent job placement rate. Overall, ninety-two percent of 
vocational program graduates are employed in their area of training, 
and employers express "a high level of satisfaction with the training 
and education of SFCC students" (Santa Fe Community College, 
1988b, p. 4). This satisfaction is reflected in the wages earned by 
SFCC graduates. The median income of the college's full-time, 
employed graduates during the period of the study was forty-eight 
percent higher than the Santa Fe per capita income (Santa Fe 
Community College, 1990). 



Student Outcomes Studies 



Regular 
reviews tied to 
outcomes- 
related goals 
are central to 
the student 
success model 



That these statements can be made and supported is a consequence 
of the college's Student Outcomes Study (SOS) program. Through 
these studies, the institution regularly reviews the progress it is 
making in meeting its outcomes-related goals. It is important to note 
that SFCC has neither an institutional research department nor a 
budget for outcomes assessment; all assessments are done by faculty 
and administrators in addition to their other duties and are funded 
through the various departments. 

Published SOS policy guidelines specify the types of outcomes that 
are to be examined on a regular basis, the frequency with which the 
studies will be conducted, and the departments that are responsible 
for completing the studies. 



Annual Studies 



• Graduate follow-up and an employer follow-up, conducted by 
guidance services 
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Success rate and withdrawal rate analysis of students in selected 
courses, conducted by the arts and sciences department 

Final grade correlational study of selected courses, conducted 
by the arts and sciences department 

Licensure exam success rate study, conducted by the business 
and management department 

Withdrawal report, conducted by the Admissions and Records 
Office 



Every two years the Admissions and Records Office also conducts a 
study of transfer students. A biennual student opinion survey is 
also conducted. The SOS also includes the option to conduct 
additional one-time, annual and biennual studies as needed. A time 
table has been adopted to ensure that each component of the SOS is 
completed in a timely manner. 



A Phase I 

program 

review 

measures 

outcomes of 

instructional 

programs 



SFCC f S Instructional Program Review 
Process 

Outcomes are also identified and measured through the instructional 
program review process. Every other year, each degree and 
certificate program receives what the institution calls a Phase I 
Program Review. This is a routine process that is a part of the 
college's instructional management system. Factors considered 
include 

r 

the program's need/mission statement 

program course enrollments for the past four semesters 

direct and indirect credit hour generation for the past four 
semesters 

demographics of students majoring in the program for the past 
two semesters 

the cumulative number of students who have graduated from 
the program 

the number of students who graduated from the program during 
the last academic year 

the availability of faculty 
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the full-time/part-time faculty ratio 



A Phase II 
program review 
indicates areas 
for 

improvement 



• the cost of the program as compared to similar programs in the 
state and region 

The review is conducted by the appropriate division head, and a 
summary is reviewed by the Dean of Instruction, 

If the Phase I review indicates areas for improvement within the 
program or if more information about the program is needed, the 
Dean of Instruction appoints a Phase II Program Review 
Committee, The committee is composed of an instructor from the 
program or the program's division head, a full-time instructor from 
another program, a student in the program, a member of the 
program's advisory committee or an external consultant, and an ex- 
officio member from the college's information services staff. 

The Phase II Program Review included assessments of the 
following: 



student satisfaction with the program 

placement rates 

the quality of the curriculum 

the credentials of the faculty 

the instructional methods used within the program 

the program's facilities and equipment 

the sequence and timing of course offerings 

student satisfaction with support services 

the continuing need in the community for the program and/or 
training 

The Committee's report to the Dean of Instruction details program 
strengths and weaknesses and recommends actions to be taken. 



Institutional Effectiveness and Student 
Success 



Outcomes that are measured directly relate to either student success 
or instructional effectiveness at the individual student and program 
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Students 
develop their 
own definition 
of success as a 
basis for 
outcomes 
assessment 



levels. SFCCs indicators of institutional effectiveness are typically 
aggregates of student and program outcomes. 

Ensuring that students successfully meet their goals is an integral 
part of the college's approach to outcomes assessment. SFCC has 
purposefully avoided developing an institutional definition of 
success. Instead, students entering the college are responsible for 
defining what success will mean for them. Faculty, staff, and 
administrators then collectively act to support the student in meeting 
these goals. 

The student success model provides the structural framework within 
which this policy operates. Under the model, student success may 
be defined as 



• improving job skills, 

• acquiring transferable credits, 



• completing a degree or certificate program, or 



• personal enrichment. 



Once a student has defined what outcome(s) will be necessary to 
achieve success, there is a mandatory advising session and 
orientation program in which guidance personnel and the student 
agree on the specific strategies that will be used to meet the 
established goal(s). For some students, the next step is course 
selection and registration. Others may first be offered special 
assistance through a variety of programs, including tutoring, career 
counseling, basic skills assessment, and financial counseling. 



Safety Checks 



Safety checks Once the initial registration period ends, the first of two "safety 
are initiated to checks" is initiated. Students who applied for admission but did not 
assist students enroll are contacted by telephone to see if their failure to enroll was 
in meeting their caused by a factor that the college could have helped them 
goals overcome. Students with remediable problems would be encour- 

aged to take advantage of the late enrollment period. Any student 
whose class was cancelled or who failed to attend the first meeting 
would also be contacted by telephone so that options available to 
them could be discussed. 



The second level of safety checks is called the "early alert" system. 
Designed as an intrusive approach to problem solving, the faculty 
and counselor collaborate on assuring early student success. 
Attention is given to attendance patterns, proper course placement, 
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learning styles, and similar issues. If absenteeism or course 
problems are noted, the instructor and counselor immediately 
intervene through direct contact with the student, referrals, and 
follow-up. 

An institution-wide check takes place for students on probation and 
for all students with a "D" or TP" at the midterm. These students 
also receive a follow-up letter from the college and axe reminded that 
their counselor is available to help them work through personal and 
academic problems. They may also be advised that they will be 
placed on academic suspension if they do not improve their grades 
during the second half of the semester* In addition to offering 
advising and counseling, the safety check invites students to attend 
successful student seminars, however attendance is not mandatory. 
These seminars are designed to provide fairly individualized 
assistance to small groups of students with the goal being to provide 
whatever is needed to help them be successful. 



Fostering Teaching Effectiveness 



Teaching 
effectiveness 
enhances 
student success 



The college sees fostering effective teaching as a primary means of 
enhancing the likelihood that students will experience success. 
Toward this end, faculty are encouraged to develop teaching styles 
that 



Staff and 

organizational 

development 

focus on 

contemporary 

issues 



provide students opportunities to engage in critical thinking, 

ensure effective communication, 

encourage a positive classroom climate, 

provide for effective and efficient time and resource 
management, and 

utilize a variety of teaching methods. 

Faculty are also encouraged to maintain and display a knowledge of 
contemporary issues and developments within their fields. The 
college employs a director of staff and organizational development to 
assist in the development, maintenance, and monitoring of these 
efforts. Through the director's office, the college offers 
approximately seventy in-service activities each year and faculty 
participation is strongly encouraged. 

Instruction is centered on a competency-based curriculum that 
ensures similarity between courses. Course-level outcomes are 
specified in the course syllabus. At this level, approaches to 
assessment vary among the courses and, to a limited extent, between 
instructors. In addition to traditional written exams, assessment 
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processes include comprehensive projects, portfolios, and exams 
conducted by panels of experts. 



SFCC f S Outcomes Committee 



Studies 
designed and 
implemented by 
an outcomes 
committee 
provide an 
additional level 
of assessment 



Most outcomes assessments above the course level are conducted 
through studies designed and implemented by an outcomes 
committee. This group is composed of faculty and administrators 
from the instructional services and student services divisions. 
Appointments to the committee are made by the college president, 
who largely relies on recommendations from the deans and division 
heads. 

Efforts are made to ensure that the committee always has a member 
with a strong research background, at least one faculty 
representative, and a balanced mix of personnel from the various 
divisions. 'The current committee is composed of . 



one instructor 



• the directors of two administrative offices 



• three division heads 



• two deans 



One of the deans is the committee chair. The president and the dean 
of instruction, who was a previous committee chair, serve as ex- 
officio members. 



The outcomes committee is responsible for overseeing the 
completion of the various mandatory outcomes studies listed above 
as well as any special studies needed to gather information on a 
point of interest. When the committee was first formed in 1983, 
almost all studies were designed independendy. Since that time, a 
research format has been established and is applied consistently to 
give a predictable structure to the reports. Reports typically involve 
hypothesis testing and/or descriptive research. When appropriate, 
they also include recommendations for action to be taken based on 
the findings. 

A study examining the effectiveness of developmental classes in 
preparing students in need of remediation for regular college level 
math and English classes is representative of other studies conducted 
by the outcomes committee. Research began with the hypothesis 
that students who took developmental classes before they took a 
college level class would have a success rate at least as high as the 
students whose first math or English class was at the college level. 
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Grades for students in the two groups were culled from the college's 
database and examined using a chi-square test The English study 
showed that students who received remediation performed as well as 
those who had not needed remediation. The math study showed that 
those receiving remediation subsequently had an even higher 
success rate than those who had not needed that service (Santa Fe 
Community College, 1989b). 



Uses of Outcomes Study Reports 



Study Once an outcomes study is completed, the results are forwarded to 

reports are the appropriate division head, program personnel, the dean of 

internal instruction, and the administrative council. The council is composed 

working of the president, the four deans, the director of planning and 

documents, development, and the director of community relations. These 

but not for persons may elect to share the study findings with other college 

public use personnel; however, the findings typically are not released to the 

public. Rather, they are considered to be internal working 

documents. 

If deficiencies or areas for improvement are uncovered during a 
study, the study serves as a basis for making the necessary 
adjustments and assessing the results. A SOS (Student Outcomes 
Study) undertaken to review the performance of Santa Fe 
Community College nursing students on the NCLEX-RN national 
nursing licensure exam illustrates how this process works. 

In 1989, eighty-five percent of the Santa Fe students who took the 
NCLEX-RN test passed it; this was the same as the national 
average. In 1990, the college's pass rate dropped under seventy- 
four percent while the national pass rose to eighty-six percent. This 
decline and the subsequent SOS study led to a series of 
recommendations designed to increase student performance on the 
test. These included strengthening the curriculum in general, 
improving specific areas of instructional weakness identified in test 
summaries, enhancing student study skills and test preparedness, 
and improving monitoring of each student's academic performance. 
Subsequent to the implementation of these changes, the college's 
NCLEX-RN pass rate rose to one-hundred percent (Santa Fe 
Community College, 1991a). 

SOS findings also influence college policy decisions. At one point, 
administrators hypothesized that students who registered for classes 
during the late registration period did so because they had not made 
adequate preparations for the upcoming semester. Such 
preparations might include obtaining financial aid, working out 
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transportation schedules, and arranging personal time to 
accommodate the demands of going to school. Students who had 
failed to make these preparations were seen as potentially high risk 
students. Given the college's emphasis on student success, it was 
thought that these students should be advised to sit out a semester 
while they made the necessary arrangements. To encourage this, 
elimination of the three-day late registration period was proposed. 

An outcomes study revealed, however, that the academic 
performance (grade point average, completion rate, and continuation 
rate) of late registrants was similar to that of all other students. The 
study also found that students who registered late for classes 
typically did not do so for the reasons anticipated. Rather, these 
students were typically signing up for classes that met on an 
irregular schedule. Because it was found that eliminating the late 
registration period could prevent students from signing up for these 
classes, the proposed policy change was not enacted (Santa Fe 
Community College, 1989a). 

Although SOS are internal documents, the college often chooses to 
report the findings to the public. Findings typically are reported 
through publications designed for key stakeholder groups including 
students, employers, receiving colleges, and the community at large. 
These publications include a variety of annual reports, recruiting 
brochures, and a news magazine produced by the college's 
community relations office. 



Implementation Issues 



Conceptual 
and 

operational 
issues must be 
addressed to 
implement the 
student 
success model 



Institutions attempting to put into operation a version of the SFCC 
/ approach to student success and outcomes assessment must address 
several conceptual and operational issues. As they evaluate these 
models from their own perspectives, practitioners typically ask five 
general questions. In separate interviews, SFCCs president and 
Student Outcomes Committee members were asked to address these 
questions. Their responses are synthesized below. 

1. Question: How does the institution ensure that the members of 
each campus unit contribute to the student success model? 

Response: Institutional environment is die most important factor in 
keeping everyone involved in the process. "It's an institutional 
philosophy," according to Professional and Technical Programs 
Division Head and Student Outcomes Committee member Shiela 
Ortego. "Student success is just what we do and everyone knows 
it. It permeates the institution It's what we are." 
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Allowing all personnel to play a role in the Student Outcomes 
Committee reinforces this philosophy. Committee membership 
changes each year, allowing most administrators and full-time 
faculty an opportunity to serve on the committee. Additionally, any 
employee can request that a study be undertaken. The factually 
based, nonpunitive approach taken by the committee, the president, 
and the institution toward these studies makes the SOS model a 
"user-friendly" process. 

2. Question: What are the guiding forces that keep the student 
success model working? 

Response: There are four principal factors involved in keeping the 
process on track. The first, according to SFCC President William 
C. Witter, is the institution's commitment to hiring people who are 
willing and able to work within the model. All persons responsible 
for interviewing job candidates stress the importance SFCC places 
on instructional quality and student success. To reinforce the 
message, the president personally interviews all job finalists before 
their hiring by the college. 

Faculty development is another factor in maintaining the student 
success model. In a reflection of Witter's belief that "everyone can 
improve," the college's staff development officer conducts over 
seventy staff development activities each year. Also, every 
instructor is evaluated in the classroom once each semester by a peer 
or division evaluator selected by the director of staff development. 
A follow-up conference is conducted and recommendations are 
discussed. The results of these evaluations, which are designed to 
encourage self-improvement, are not reported to the president. 

The third factor in ensuring implementation is the president himself. 
In this regard, Witter sees himself as both a coach and a cheerleader. 
As he sees it, his job includes "firing up people all the time" and 
reminding them of the importance the institution places on student 
success. 

Faculty attitudes are another mechanism that keeps student success 
at the forefront of institutional philosophy. According to an 
instructor on the outcomes committee, "We get the message to 
support student success from above, but we aren't being forced. 
Everyone is concerned that their program is doing well, that we're 
preparing [students] for the next level, that we're preparing them for 
jobs. It's a concern of the faculty in general and the administration 
in general." 

3. Question: What factors should be considered when trying to 
establish a student success model at an existing institution? 

Response: According to Marsha Drennon's, Student Outcomes 
Committee member and Director of Planning and Development, the 
most important factor is that the model focus on student outcomes 



Four principal 
factors keep the 
student success 
model on track 
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rather than on faculty performance. While the institution does act to 
ensure that faculty performance meets established standards, the 
student success model is not a part of that effort. Thus, it is not 
threatening to the faculty. 

Faculty ownership, according to Shiela Ortego, is another crucial 
aspect of the model. John Pantano says, "Even though it is a 
mandate, we are always asked for our input. If we were forced into 
this model, we would not be so happy with it. But we had input 
into what we wanted to do from the beginning." 

4. Question: How can administrators win over skeptical faculty? 

Response: Shiela Ortego believes that the first steps should be to 
emphasize that the process is not punitive and to reassure the faculty 
that it does not place their jobs in jeopardy. According to Witter, 
SFCC maintains a very low employee turnover rate. While this 
relates in large measure to the caie taken to make proper choices 
during the hiring process it also documents that student outcomes 
studies are not used as a rationale for dismissal. 

Ortego suggests that a college just developing the process should 
implement it slowly and with a maximum amount of faculty 
involvement. She cites SFCCs decision to pilot test the process 
with willing faculty before it was implemented across the board as 
an important factor in the program's success at the college. When 
used by SFCC, this approach gave other faculty a chance to hear 
and see positive benefits of the program before they became 
involved with it. 

Marsha Drennon confirms that the system's benefits tend to be its 
greatest selling points. As people are involved in or surrounded by 
the studies, they see that the process is a proactive tool that they can 
use to improve their own performance. According to John Pantano, 
SFCCs SOS model did not gain acceptance overnight. The math 
and English faculty were the first to buy into it; their success in 
documenting strengths and identifying and resolving problems 
encouraged other departments to buy into the process as time went 
on. Despite their nonpunitive emphasis, Pantano points out that the 
studies must not "whitewash" areas for improvement. In order to be 
of value to faculty, the studies must strike an objective balance 
between these two extreme positions. 

5. Question: How can an outcomes committee develop and retain a 
positive working relationship with the rest of the college 
community? 

Response: One key is that each department retains responsibility for 
actually conducting its outcomes studies. According to Shiela 
Ortego, this helps make the process more meaningful to the 
departments and increases the likelihood that they will use the data. 
Marsha Drennon points out that studies are done to support 
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auricular change, to meet grant requirements, or for other internal 
purposes. The studies are not done as an academic exercise or 
solely to meet committee needs or external accountability 
requirements. Instead, the committee assists with and reviews 
studies that would be needed even if the SOS process did not exist. 
Deriving outcomes data from department studies also helps keep 
outcomes from being a "mystery." 

Building on the notion that outcomes studies should be demystified, 
John Pantano suggests that two-way lines of communication 
between the college community and the committee must be 
established and maintained Rotating committee memberships helps 
facilitate communication, as does disseminating findings among 
department faculty for review before the report is passed onto 
administrators. This helps ensure that reports are accurate and 
allows front-line personnel to be aware of and assume ownership of 
the results of the studies. 



Exporting the Process 



SFCC's 
success model 
is data-based 
and issues- 
oriented but 
still 

nonthreatening 
to faculty and 
staff 



Key points in SFCC's student outcomes model are that its studies 
arc 

on-going, 
brief, 
practical, 
issues-oriented, 
data-based, and 

relevant to the daily work of educators. 

Further, the model operates within an environment purposefully 
. constructed to: 

• be nonthreatening, 

encourage full faculty and staff participation, and 

encourage effective instruction. 

As a consequence of these efforts, which should be replicable at 
other institutions, the student success model has been fully 
integrated into daily operations at SFCC. 
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Chapter 4 

Value-added Assessment 

C. Michael Harmon 



Overview 



In this chapter 



Although it is a contemporary topic of discussion, value-added 
assessment has been applied sporadically in American higher 
education for nearly two decades. The concept of value added 
originates not in education but in economics, where it is used to 
describe "the value of [a company's] outputs minus what it buys 
from other firms" (Weiss, 1981, p. 385). In other words, value 
added "is the difference between what a firm sells its product for and 
what it pays for the immediate materials or good it processes../ 1 
(Waud, 1980, p. 96). 

This chapter examines the meaning of value-added assessment and 
examines the means by which three institutions are measuring the 
value added to students by their programs. Readers will see the 
potential connection of two previous chapters in this book, Total 
Quality Management (TQM) and Assessing Student Success, to 
assessing value added. 

This chapter discusses the following topics: 



Topic See Page 
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Measuring the Value Added by Education 
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Early Efforts in Value-Added Assessment 
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Current Issues in Value-Added Assessment 
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The Relationship Between Value-Added and Outcomes 
Assessment 
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The Inadequacy of Measuring Only Growth 
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Value-Added Assessment as a Tool for Change 
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Pretest and Posttest as a Measure of Learning 
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Assessment at Alvemo College 
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Measuring the Value Added by Education 



"Value-added 

education 

examines 

changes in 

students' 

performance 

over time." 



Education is fundamentally a value-added process. Although it 
states its goals in such terms as "helping students develop their 
inherent potential," schooling is essentially "an act of processing 
people in some way for the purpose of adding value" (Connor & 
Lessinger, 1983, p. 2). McMillan (1988, p. 564) observed, "Value- 
added education examines changes in students' performance over 
time. Students are assessed for entering competencies and then 
reassessed following the completion of appropriate courses or 
experiences," The challenge, then, is to define and measure the 
value that educational institutions add to their students as can be 
evidenced, ultimately, be program and student outcome. 



In its simplest and most traditional form, value-added educational 
assessment involves the administration of a pretest, exposure to a 
learning experience, and administration of a posttest. This is one 
method of measuring the gains — or possibly losses — that students 
experience as a consequence of involvement with an educational 
program. At the Terry Campus of the Delaware Technical and 
Community College, located in Dover, an assessment process that 
utilizes the pretest and posttest design is being developed and 
implemented 

In a process coordinated through a newly established Assessment 
Center, Delaware Technical Community College uses a pretest to 
measure the abilities of incoming students. The Center is charged 
with helping the college answer three questions: (1) What should 
students learn? (2) How well they are learning it? and (3) How does 
the institution know how well they are learning it? To carry out its 
mission, the Center first pretests students to determine appropriate 
class placements. The Center then tracks students and measures 
academic gains through the use of mid-course assessments and exit 
tests. Counseling and support services are also provided through 
the Center. 

Earl Roberts, Title III Director at the Terry Campus, describes the 
college's process as being "heavily based on Deming's philosophy." 
It is, according to Roberts, designed to look for incremental 
institutional improvement. That is to say, the value-added 
assessment process used at the college measures student 
performance as an indicator of institutional effectiveness. This 
effort to tie institutional effectiveness to the ability to meet client 
(i.e., student) needs is, as Bragg noted in Chapter 2 of this 
document, an integral part of the TQM approach. 

Pretest and posttest is generally recognized as the most common 
means of measuring value added. It has been suggested, however, 
that pretest and posttest is not the only way that value added can be 
assessed (Belcher, 1987; Hanson, 1988; Turnbull, 1987). 
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The most 
common way to 
measure the 
value added by 
education is to 
pretest and 
posttest 



Removing this restriction opens the possibility of using vahe-added 
assessments in ways that go beyond institutional effectiveness. As 
Pascarella (1986, p. 75) observed, "If we are willing to accept 
value-added as a potentially important approach to the assessment of 
student outcomes, then it behooves us to consider ways In which the 
methodology of the approach might be enhanced and sharpened" 

There are certain prerequisites to being able to successfully measure 
the value added by higher education. Fincher (1985) has suggested 
that value-added assessment cannot hope to be a viable methodology 
unless an institution first agrees on the outcomes that are to be 
assessed, develops appropriate assessment mechanisms, and 
focuses instruction to address specific objectives. Elsewhere in this 
book, approaches related to outcomes conceptualization (see Chapter 
5) and performance assessment methodology (see Chapter 7) are 
discussed* These approaches provide some of die tools for the 
implementation of value-added assessment* 



Early Efforts in Value-Added Assessment 



Two institutions, Northeast Missouri State University (NMSU) and 
Alverno College, were early leaders in die effort to apply value- 
added assessment in higher education. The objectives of the two 
institutions and the approaches they chose were, however, quite 
different. In 1974, the faculty and administration at NMSU set out 
to devise a system that would allow them to simultaneously 
"improve the quality of students* learning . . . , husband scarce 
resources, and ... be accountable." Value-added assessment was 
seen as a "unified response to these challenges" (Northeast Missouri 
State University, 1984, p. 6). 

The faculty at NMSU designed their value-added approach to 
measure the effectiveness of their institution. Early on, this 
involved a pretest and posttest approach to assessment The goal 
was to discern whether the institution had been effective in its 
educational program. While the institution still relies on value-added 
assessment, it is generally moving away from a predominantly 
pretest and posttest design. Today, their approach is evolving to 
include performance assessment (Rector, 1992). Portfolios and 
capstone experiences are used in various programs across the 
campus (Young, 1992). 

The faculty at Alverno College in Milwaukee, Wisconsin, devised 
an approach to value-added assessment that serves purposes quite 
different from the original intent of NMSU's process. Alverno 
College has stated that, "We are committed to assess what each 
student is learning in every course in our curriculum. We are also 
committed to assessing student competence outside the classroom" 
(Alverno College Faculty, 1985, p. 4). Because the college focuses 
its process on documenting student development rather than 
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demonstrating institutional effectiveness, the faculty rejected the 
pretest and posttest design from the start, preferring instead to 
develop and validate performance assessment measures (Loacker, 
1992). 

As their longstanding commitment to employing their respective 
approaches attest, the faculty at both Alverno College and NMSU 
believe in value-added assessment The faculties at some other 
institutions have also incorporated value-added approaches into their 
daily operations. (See, for example, Erwin, 1991, p. 20-23.) 
However, value-added assessment has largely been ignored by 
America's postsecondary education system (Kirst, 1991; Smith, 
1991). 



Current Issues in Value-Added Assessment 



Differences of 
opinion exist 
about the 
meaning and 
implications of 
value-added 
assessment 



As early experiments with value-added assessment gained attention, 
so too did the diversity of opinion as to its meaning, implications, 
and limitations. Hanson drew a distinction between outcomes-based 
and value-added assessment, suggesting that "the focus of student 
outcomes assessment is the 'what' of education, and the focus of 
value-added is on the 'how'" (1988, p. 53). Therefore, according 
to Hanson, assessment must be partitioned into distinct and 
potentially competing parts rather than viewed as complementary to 
the whole system, as is done by the TQM and student success 
approaches previously discussed in this book. Following Hanson's 
logic further, however, leads to consider value added to be a 
longitudinal approach to answering the question, "What kinds of 
students change in what kinds of ways when exposed to what kinds 
of educational experiences?" (Hanson, 1988, p. 53). 

Belcher (1987) indicated that Hanson's concern about the limitation 
of value-added assessment was shared by others. Recurrent themes 
identified by Belcher were that 

value-added assessment emphasizes learner growth, which is 
not the only important factor for educational programs. 
Growth, while important, does not imply the attainment of 
competence, for example. 

• value-added assessment causes institutions to change the way 
they operate by causing institutions to overemphasize measured 
skills to the detriment of equally important unmeasured skills. 

• value-added assessment is often seen as following a 
standardized pretest and posttest design that may be based on 
inaccurate assumptions about the nature of learning. 
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The Relationship Between Value-Added 
and Outcomes Assessment 



Outcomes- 
based and 
value-added 
assessment can 
reveal "the 
level of 
developed 
competency or 
talent of the 
graduates of an 
institution" 



The 

institution's 

mission 

statement 

ultimately 

drives what is 

assessed by the 

institution 



Although Hanson and Belcher contended that outcomes-based 
assessment and value-added assessment can be two very different 
things, in practice the difference may be nothing more than a 
question of what types of change in students is thought to be of 
value* What is valued by stakeholders is an especially important 
consideration in educational contexts in which workforce 
preparation is a primary goal. In these situations, "the employer is 
really only interested in outcomes — that is, the level of developed 
competency or talent of the graduates of an institution" (Astin, 1987, 
p. 94). The degree of change in competence is not likely to be 
important to an employer if a student has not met specified 
performance levels. 

Perhaps looking at the issue from this perspective, others have seen 
a relationship between outcomes and value-added assessment* 
McMillan (1988, p. 565) observed that "experience with value- 
added education models over the past few years suggests that the 
emphasis on documented improvement of students' knowledge, 
skills, attitudes, and other outcomes has had a positive effect on 
higher education," 

The question of value is one that must ultimately be addressed in an 
institution's mission statement (Astin, 1987). In some cases, an 
institution may wish to measure changes that are not captured by 
outcomes statements. In other cases, though, the acquisition of 
certain pie-specified knowledge, skills, and attitudes is precisely the 
value that an institution seeks to add to its students. In these 
instances, where the value added is the acquisition of specific skills 
gained at the course level rather than those associated with degree 
completion, value-added assessment and outcomes assessment are 
congruent efforts. In either event, Belcher (1987, p. 32) points out 
that "measuring improvement does not replace the possibility of 
setting a floor by [establishing] exit standards." 



The Inadequacy of Measuring Only Growth 



That value-added assessment measures growth is an unassailable 
position* Within this context, Hanson (1988) has identified five 
conceptual issues that need to be addressed in a way that makes 
sense for each particular institution. These issues include 

• What should be measured? 

• When should the measurement occur to accurately reflect 
student growth? 
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♦ Will the assessment indicate how the student changed? 



Measuring 
growth is of 
critical concern 
for 

postsecondary 
students who 
move into the 
job market 
before 
completing 
programs 



♦ Can effects found by assessment measure both direct and 
indirect factors that lead to change? 

• Should the assessment measure both direct and indirect factors 
that lead to change? 

The question of what should be measured is central to establishing 
any outcomes assessment program, including value-added 
assessments. Some have criticized value-added assessment for 
placing too much emphasis on growth rather than on the attainment 
of competence (Manning, 1987). Employers, as was discussed 
previously, might have little or no interest in whether an employee 
has skills beyond those needed for employment Indeed, from the 
employers perspective, the addition of talent beyond this level might 
"be superfluous or possibly irrelevant" (Astin, 1987, p. 94). 

According to Turnbull (1987), it is important to measure both 
growth and terminal competence. This is of critical concern in those 
instances where postsecondary students enroll in technical curricula, 
gain skills through die completion of certain career-relevant skill 
development courses, and move into the job market without 
completing a degree or certificate program. Here, failure to address 
both aspects of learning can lead to a situation where an institution 
that is highly successful in producing a competent work force can 
appear to be ineffective because of a high non-completion rate. 

Providing measures of both growth and terminal competence is well 
within value-added assessments capability, countering the 
perspective held by Hanson and others. Therefore, value-added 
assessment enables an institution to accurately state its effectiveness 
from institutional and program perspectives. Establishing 
benchmarks at critical points in students' educational programs and 
assessing students in relation to them is one way to meet this goal. 
Taking this approach will move institutions toward establishing 
meaningful performance measures and standards and meeting the 
mandates of Perkins n. (See Chapter 1 for a discussion of current 
federal mandates). 



Value-Added Assessment as a Tool for 
Change 



Change is an inevitable part of life; keeping up with the rest of the 
world requires that we, too, change. Institutions are required by the 
federal government, roughly two-thirds of the states, and all 
regional accrediting agencies to attend to assessment issues. (For 
further discussion, see Chapter 1 on assessment policies.) This 
emphasis mandates changes in practice at many institutions. The 



56 



9 

ERIC 



73 



Value-added 
assessment helps 
make informed 
decisions that 
can guide an 
institution 
through change 



Value-added 
approaches 
engender 
change at the 
program level 
to ensure 
students are 
learning what 
they need to 
know 



issue is not whether there will be change: it is a given. It is, instead, 
an issue of determining the basis for the information used to make 
decisions that guide institutions and programs through periods of 
change. Value-added assessment provides an opportunity to make 
those decisions based on information about educational quality 
(NMSU, 1984), 

Astin points out that higher education institutions have traditionally 
rated themselves on the reputation and/or resources that they have 
acquired over time. For those colleges who consider reputation to 
be of importance, excellence is measured according to their ranking 
in comparison to peer institutions. For those who consider 
resources to be of paramount importance, factors such as student 
admission requirements, faculty productivity, physical plant 
facilities, and financial resources are indicators of excellence. There 
is no guarantee, however, that excellence in terms of ranking, or 
resources will lead to educational excellence. Further, some 
institutions with reputations for excellence in neither area still 
manage to provide very effective undergraduate education (Astin, 
1987). 

An effective educational program is the result of many factors, 
among them good decision making at a number of levels. The 
faculty at NMSU have written that, 

Sound decisions require judgement and facts. When 
an institution knows the facts about its impact on 
student learning, it is in a position to evaluate courses 
of action in terms of their potential influence on how 
much students learn. Decisions based on quality are 
sometimes remarkably different from decisions based 
on credit hour production. Sometimes, too, 
improvements are made at the expense of some 
cherished old myths (1984, p. 48). 

Earlier in this chapter, it was observed that the determination cf what 
is of value in an institution should be directly related to the 
institution's mission statement The subsequent construction of an 
assessment program requires that clear and assessable educational 
goals and objectives be established (Erwin, 1991; Pascarella, 1986). 
This in itself is quite likely to engender change within the institution 
at the program level as instructors adjust their teaching to ensure that 
students arc being taught and are learning what they need to know 
(NMSU, 1984). 
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Pretest and Posttest as a Measure of 
Learning 



Alternative 
assessments 
such as senior 
theses or 
capstone 
projects can be 
used to measure 
the value added 
by an 

educational 
experience 



In both popular and professional literature, value-added assessment 
has been closely linked with a pretest and posttest design. Hartle 
(1985) defined value-added assessment as a pretest and posttest 
process designed to measure both general education and skill 
development. McMillan (1988) explained that the pretest and 
posttest design allows for the assessment of changes in competence 
over time. 

Discussing the many statistical flaws inherent in simple pretest and 
posttest assessment, Hanson (1988) points out that this model may 
not be a particularly good way to conduct value-added assessment 
He recommends instead that causal analysis and hierarchical linear 
regression models of data analysis be used to fully investigate the 
extent and nature of change rather than relying on a simple 
assessment that may include confounding factors. 

Belcher (1987) also pointed out that while pretest and posttest is one 
way to measure the value added by an educational experience, it is 
not the only design available. She advocates that senior theses, or 
presumably other capstone projects, be retained and used as 
benchmarks for measuring "varying levels of acceptability" (1987, 
p. 34). In instances where institutions feel a need to use 
standardized tests for pretest and posttest measurement of skill 
improvement, Belcher (1987) proposed that a four-step process be 
used: 

• Administer an entry-level test and use scores for course 
placement 

• Determine which elements of the curriculum should affect the 
skill levels measured at entry and collect information on these 
elements. 

• Select an exit test, either the same one used at entry or a more 
difficult one. 

• Conduct early statistical analysis to determine the extent to 
which entry skills and curricular elements predict exit skills. 

If a commercially prepared test is used, it can usually be assumed to 
have been validated for a specific purpose. However, it remains the 
institution's responsibility to ensure that it is used for the purpose 
for which it was validated. 

Alverno College breaks away from the pretest and posttest design of 
value-added assessment by using narrative evaluations that are 
integrated into the learning process to measure both student growth 
and competence attainment That these assessments are qualitative 
does not diminish their appropriateness. As Patton (1988, p. 75) 
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Determining the 
validity of a 
measure 
requires 
understanding 
the educational 
assumptions 
made by the 
institution 



pointed out, M [F]or particular outcomes no acceptable, valid, and 
reliable quantitative measures exist . . . fT]he state of the art in 
social science measurement is such that a number of desirable 
outcome measures still elude precise measurement" These 
outcomes probably include creativity, self-esteem, and others that 
are identified as being important to educators, students, employers, 
and reform leaders. 

Reliance on qualitative descriptions of student performance in no 
way reduces the importance of establishing the validity of the 
process. The same internal and external threats to validity that 
Campbell and Stanley identified for experimental and quasi- 
experimental research also apply to qualitative assessment (Lincoln 
&Guba,1985). 

How is validity established? House (1980, p. 249) observed that 
"as a minimum validity claim, one would expect that the evaluation 
be true. There are different ideas and methods for arriving at the 
truth." A valid instrument truthfully represents what it says it 
represents. Mentkowski and Rogers (1988) contend that any means 
of determining validity must be appropriate to the educational 
assumptions made by the institution at which the particular 
instruments are being used. There is, then, no magic formula for 
developing institution-specific responses to the challenges of 
instrument validation. 

It is readily apparent that validating performance assessments can be 
a difficult task. However, vocational educators have a long tradition 
of measuring student performance through just such measures. 
Recent research conducted by Mabry (1992) (see Chapter 7) 
suggests the issue may not be whether such assessments can be 
conducted in ways that are valid and reliable on a large scale but 
whether they yield useful information on an individual basis. To 
address this question, well-established performance assessment 
approaches used at Alverno College are discussed in the remainder 
of this chapter. 



Assessment at Alverno College 



Alvemo College is a private, Catholic women's liberal arts college at 
which assessment is an important part of academic life. The 
Alvemo faculty report that "students vary in their responses to this 
first encounter (with the college's assessment program), but they all 
testify to its impact and complexity. . . . Even with a prior visit to 
the campus, one junior at the college noted, It was still really 
confusing. You have to experience it to really understand the value 
of tf" (Alvemo College Faculty, 1985, pp. 38-39). The following 
narrative documents their approach to assessment and provides 
readers with just such an "experience." 
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Alverno focuses It is important to note that while Alverno's assessment process 
a "value-added" measures the growth that occurs in students over time, there are 
approach on sharp distinctions between this college's student assessment process 
student and the pretest and posttest notion of value-added assessment 

assessment described by Hartle (1986) and McMillan (1988). According to 

Assessment Committee Chair Georgine Loacker (1992), and 
important distinction is that Alverno uses multiple assessments 
across time not only from an institutional research perspective but 
from the student's perspective, to enable her to trace the patterns of 
her development of ability and to demonstrate her progress. 

The generic application of value-added derives from the college's 
initial assessment of students upon entry into the college and the 
continuous use of multiple performance assessments to document 
student growth (Loacker, 1992). The initial assessment typically 
takes place on a single day, although some students are permitted to 
spread out the process to accommodate personal needs. During the 
course of the day, students are assessed and assess themselves in 
six areas: writing, speaking, listening, reading, computer literacy, 
and quantitative skills. Writing and speaking abilities are assessed 
through a letter-writing exercise and a videotaped speech. Skills in 
the four other areas are assessed through responses given to a series 
of open-ended questions (Alverno College Faculty, 1985). 

One current Alverno student described the day-long assessment 
process in this way: 

"One day at the beginning, a series of tests was given. A 
speaking section, a writing section, math; they hit 
everything. They want to see where you're at, what you're 
comfortable with in terms of writing and speaking and so 
forth. Nobody wants to be placed in a class that they're 
going to be uncomfortable with or nervous with* They want 
to see where your abilities lie and put you in a comfortable 
environment, a class where you'll be comfortable and that 
you're ready for because the classes build on one another." 

One of Alverno's most widely recognized contributions to student 
outcomes assessment is the use of a matrix that defines abilities to be 
developed and demonstrated by students. In order to graduate, each 
student must fully meet the criteria in each of the forty-eight matrix 
cells (Figure 1). The six skill levels in each of the eight areas of 
ability are sequentially airanged so that developing ability in one cell 
is a prerequisite for satisfying the requirements of the next cell. 
While students must demonstrate mastery of each ability, it can be 
demonstrated in a number of ways and in a variety of courses 
depending on the program of study. The completion of these cells 
documents student growth in each of the eight ability areas. 
Instruction is tied to explicitly stated criteria. One student who had 
transferred to Alverno from another college observed, "At the other 
college, they didn't have very specific criteria — read this book and 
write an essay on it Here, I was assigned to read a book and given 
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specific things to look for and write about The criteria helped me 
know what I should be doing." 



Continuous 

multiple-method 

assessment 

develops a 

record of 

student 

performance 



During any course of study at Alverno, assessment techniques 
include written tests, verbal feedback from the instructor, peer 
assessment, and self-assessment. Through the continuous, 
multiple-method assessment of student performance, the College's 
Assessment Center develops a comprehensive record of 
performance for each student It is the ability to use these records to 
document student learning over time that leads Alverno college to 
being a pace-setting user of value-added assessment 



Summary 



In it's most commonly recognized form, value-added assessment 
utilizes a pretest and a posttest to measure student growth over time. 
While standardized tests have traditionally been seen as the preferred 
means of conducting such assessments, other assessment methods 
can also be used. These assessments can take any of several forms 
and their selection should be driven by the mission and goals an 
institution sets forth for itself and its programs. Although less 
commonly applied, value-added assessment can be determined 
through a pretest and continual assessments throughout a student's 
program. This approach enables an institution to use multiple 
methods of assessment and to document student learning over time 
on a continuous basis. 
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Chapter 5 

Concept Mapping 

Thomas E. Grayson 



Overview Once an institution commits to assessing outcomes and determining 

the value added to each student's repertoire of skills and knowledge, 
" a methodology for conceptualizing outcomes is needed. A new 
approach that offers promise for outcomes assessment is concept 
mapping. This process enables diverse groups of practitioners to 
describe how their programs work and then, based on that collective 
wisdom, articulate how these programs influence outcomes for 
participants. 

This chapter describes concept mapping, a practical approach to 
articulating how programs work and identifying program outcomes. 
Concept mapping is a bottom-up approach and results in a pictorial 
representation of what practitioners think about how their programs 
produce outcomes. By using a case study in concept mapping to 
conceptualize vocational education outcomes at Black Hawk College 
in Moline, Illinois, this practical approach to outcomes identification 
is illustrated. 

In this chapter This chapter presents a discussion of the following topics: 



Topic See Page 



Concept Mapping: Its Theoretical Bases 66 

The Three-Phase Model 67 

Purposes of Concept Mapping 68 

Concept Mapping Methodology 69 

The Concept Mapping Process 69 

A Case Study in Using Concept Mapping to Identify 75 
Outcomes for Vocational Education 



The author gratefully acknowledges the contributions of DebraD. Bragg and C. Michael 
Harmon, who were members of the research team conducting concept mapping at Black 
Hawk College in Moline, Illinois. 
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Concept Mapping: Its Theoretical Bases 



A picture of 
practitioners 9 
thinking is the 
essence of a 
concept map 



Concept mapping has been described as a type of structured 
conceptualization process that combines group process activities 
with multivariate analytical techniques to produce a visual 
representation of the group's thinking (Linton, 1985; Trochim, 
1989b)* In concept mapping, ideas (i.e., concepts) are described 
and the interrelationships between them are articulated. The 
concepts and their interrelationships are then represented in the form 
of a visual map. 

A concept map is a pictorial representation of the 
group's thinking which displays all of the ideas of 
the group relative to the topic at hand, shows how 
these ideas are related to each other and, optionally, 
shows which ideas are more relevant, important, or 
appropriate (Trochim, 1989a, p. 2). 

Concept mapping is a clearly defined process (i.e., the process has a 
clear beginning, middle, and end) which 

♦ has a specific focus on a topic or construct of interest 

♦ involves one or more participants in the generation of ideas 
regarding the focus of interest and 

♦ is designed to produce a graphic representation in the form of a 
concept map of the participant stakeholders' ideas and how 
those ideas are interrelated. 



Concept mapping, as defined in this chapter, grew out of a general 
model of structured conceptualization, which was developed by 
Trochim and Linton (1986). These researchers first used concept 
mapping to describe program theory by understanding the causal 
links practitioners made between the operation of a program and its 
intended effects on outcomes. Implementation of this model yielded 
a pictorial representation of the ideas of participant stakeholders. 

Practitioners of vocational education prescribe particular courses of 
action to achieve certain vocational education outcomes. In doing 
so, they act as theorists. Behaviors, interpretations, and responses 
to situations represent their view of how vocational education 
programs are intended to function. Kolb argues, "Theory, an 
understanding of how program elements fit together and how 
programs influence outcomes for participants, can and should be 
built from the ground up" (1992, p. 26). He contends that 
knowing how programs work is essential to assessing outcomes and 
improving programs. 
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The Three-Phase Model 



As is shown in Table 1, there axe three general phases in the model: 

1. Generating the conceptual domain 

2. Structuring the conceptual domain 
3* Representing the conceptual domain 



Table 1 

General Model for Conceptualizing Program Theory 



1. 


2* 


3. 


Generation of 


Structuring of 


Representation of 


Concepts 


Concepts 


Concepts 


Concepts are created 


Relationships between 


The structured set 


out of thoughts, 


and among concepts 


of concepts is repre- 


ideas, intuitions, 


are defined or estima- 


sented verbally, 


theories, and 


ted 


pictorially, or mathe- 


problem 




matically* 


statements. 







Note: Adapted from Trochim & link 1986 



In the first phase of the model, the conceptual domain is generated; 
concepts are created out of thoughts, intuitions, ideas, theories, or 
problem statements* Each concept is distinguishable and can be 
expressed as a word, phrase, sentence, or other text unit These 
words or phrases form the basic unit of meaning for the structured 
set of concepts that are represented verbally, pictorially, or 
mathematically* 

The second phase is the structuring of concepts; it involves a 
process that defines relationships between and among the concepts* 
These relationships, when defined, often result in an interpretable 
and understandable arrangement of concepts. 

The third and final phase in conceptualizing program theory requires 
that the concepts be represented in some fashion* Conceptualization 
of concepts may be represented verbally, pictorially, and 
mathematically* 
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Purposes of Concept Mapping 



Concept 
mapping is a 
method for 
structuring 
participant ideas 
in an objective 
form 



Concept maps 
can display 
major program 
components as 
a guide to 
determining 
outcomes 



Concept mapping can provide a methodology for articulating and 
structuring participant stakeholders' ideas, thoughts, or concepts in 
an objective form — the concept map. Such a representation is 
extremely useful in developing program theory, identifying program 
outcomes and providing feedback to make program improvements. 
Concept mapping can also be used to guide policymakers, 
managers, program designers, and administrators in setting policy, 
conducting operational and strategic planning, and facilitating 
program planning and implementation. 

Concept mapping methodology has roots in theory development, 
particularly learning theory and program theory development. 
Learning theorists have used concept mapping to describe how 
knowledge is constructed (Novak & Gowan, 1984), and program 
theorists have used concept mapping to describe program 
components, processes, and outcomes. Program theory can provide 
"a plausible and sensible model of how a program is supposed to 
work" (Bickman, 1987, p. 5), and the concept mapping process 
offers a way to conceptualize and visually represent the theoretical 
underpinnings of the program under study (Chen, 1990; Trochim, 
1989b). 

Kurt Lewin once said, "[Tjhere is nothing so practical as a good 
theory" (1951, p. 169). Lewin was suggesting that theo7 and 
practice should be integrated and that the best use of theory is to 
facilitate understanding the field or subject under study. Grayson 
(1992), Kolb (1991), Linton (1985), Marquart (1990), and 
Valentine (1989) have all used concept mapping to create a 
framework to plan, implement, and evaluate a wide range of 
programs. 

Understanding the theoretical underpinnings of a program helps to 
understand how programs can be expected to work before the 
attempt to evaluate them is initiated. Knowing how programs work 
can aid cvaluators in determining what to measure in a program and 
what characteristics of a program are most important. Knowing 
how programs work can also help policy makers and program 
planners and managers make informed decisions regarding program 
policy, planning, implementation, operations, and improvement As 
noted by Chen (1990), when a particular program is specified, 
constructed, and implemented appropriately intended outcomes are 
highly likely and the credibility of the program is strengthened. 
Concept maps can display the major components of a program and 
. can be used as guides to determine program outcomes and to assess 
and improve educational programs. Concept maps can also be used 
to pictorially represent relationships among program components 
and to show how these components are linked to outcomes. 



68 



ERLC 



/ rt 

'O 



Concept maps 

can influence 

program 

implementation 

and ultimately 

program 

improvement 



Kolb (1991) suggests another role for concept maps in identifying 
program theory. Identifying program theory can serve "as feedback 
to inform and enlighten program practitioners, from whom program 
theories-in-use originate and [who operate at] the level at which 
program implementation and program improvement ultimately must 
take place" (Kolb, 1991, p. 18). Program theory illustrated through 
a concept map can also provide feedback to inform and enlighten 
those who created and funded a program. This type of information 
can lead to improvements in policies that govern funding and 
implementation. 



A variety of 
qualitative and 
quantitative 
methods are 
used in concept 
mapping 



Concept Mapping Methodology 

Concept mapping uses a variety of qualitative and quantitative 
methods in its procedures. Concept mapping employs 

• small group processes to identify ideas on a specific topic of 
interest (e.g., brainstorming, nominal group techniques, focus 
groups, and Delphi) 

• sorting procedures whose purpose is to group ideas into piles 
that make sense to the participant stakeholders 

• Likcrt-typc rating scales to rate the priority or importance of 
ideas on a dimension of interest 

• multivariate statistical analyses on the sorted and rated ideas in 
order to create interval-level concept maps that visually 
represent the topic of interest 

After the maps have been created, participant stakeholders are again 
involved in their interpretation with attention keenly paid to ways the 
maps can be used. 



The Concept Mapping Process 



A plan must be 
developed and 
facilitators 
selected prior to 
conducting 
concept mapping 

Selection of the facilitator is crucial. The facilitator must be familiar 
with concept mapping technology and, ideally, have substantive 



It is crucial to develop an overall concept mapping plan prior to 
conducting the concept mapping process. The organizational context 
needs to be identified and a facilitator of the entire concept mapping 
process must be designated. The goals and purpose of a concept 
mapping process must also be clearly defined; careful selection of 
participant stakeholders must be made. 
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The facilitator 
must be 
masterful at 
leading small 
groups 



knowledge of the context under study. Most importantly, the 
facilitator must be masterful at leading small group processes. The 
role of the facilitator is to guide persons involved in initiating the 
concept mapping process toward reaching their intended goals and 
purpose. Skills in facilitating a diverse stakeholder group to reach 
consensus are essential. The facilitator manages the process, 
whereas the participant stakeholders determine the content, interpret 
the maps, and use the results. 



Based on work by Grayson (1992), Linton (1985), Trochim and 
Linton (1986), and Trochim (1989b), the concept mapping process 
involves six basic steps. 



Step Primary Activity Tasks 



1 

1 




• 
• 
• 
• 


Sftlttit oatticinants 
Develop a focus statement 
Determine a rating focus 
Schedule sessions 


L 


Generation of ideas 


• 
• 


oeneratc loeas 
Reach consensus on the 
meaning of statements 
Eliminate redundant 
statements 


3 


Structuring of ideas 


• 
• 


Rate each statement based on 
the rating focus developed in 
Step 1 

Sort and label the statements 


4 


Computation of maps 


• 


Analyze data and create maps 


5 


Interpretation of maps 


• 

• 
• 


Prepare maps and materials 
for interpretation 
Interpret maps 
Consider ways maps can be 
used 


6 


Utilization of maps 


• 


Make plans to put findings 
from concept mapping into 
action 
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Step 1 - Preparation 



Selection of 
participants 
is an 
important 
first step 



There axe four major tasks in this preparation step* First, participant 
stakeholders must be selected* All participants are considered 
experts, and the number of participants should be large enough and 
diverse enough to ensure that a wide range of ideas are represented 
At the same time, the group should be small enough to be 
manageable and effective in group discussion sessions* 

My experience indicates that a group of approximately twenty is 
manageable yet large enough to generate a range of ideas* Care 
should be given to how participants are selected, whether selected 
purposively (Patton, 1980; Stake, 1985) or by random sampling 
methods* When purposive sampling is used, it is essential to 
consider selection of participants who 

• have experience participating in small group processes 

• can articulate their thoughts clearly and assertively in a group 
setting that involves people representing varying roles and 
responsibilities 

* will accurately represent the opinions of the stakeholder group 
they represent 

♦ have the analytical ability to categorize and prioritize large 
amounts of information quickly 

Again, my experience suggests that most individuals have little 
difficulty participating in the process, but some individuals can be 
overwhelmed by it* Therefore, careful selection of participants is 
essential* 

Second, a specific focus statement for conceptualizing a domain of 
interest must be developed* Clearly, the focus statement depends 
upon the purpose of the concept mapping process* The statement 
should be clearly and precisely stated* In selecting and refining a 
focus statement it is important to try it out on others* See if it 
generates the kinds of ideas you are seeking* This actively helps to 
eliminate any confusion about purpose and ensures a smooth 
beginning to the small group activity. 

For example, if the purpose is to conceptualize a particular program 
or treatment for the purpose of identifying outcomes, then the focus 
statement might read, "Generate statements (short phrases or words) 
which describe what you think the program or treatment ought to 
be*" This type of broad statement will result in the generation of 
many ideas on what the program or treatment ought to be* Later, 
when the conceptualized program is displayed in the form of the 
concept map, the map can be used to generate or identify specific 
outcomes* 
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Third, if a relative sense of importance or priority for the 
conceptualized components of the program or treatment is needed, a 
specific focus statement for rating ideas should be determined. 



Sample Rating Focus 



Rate each brainstormcd statement on a 1 to 5 priority scale where: 

1 = vay low priority 

2 = low priority 

3 = moderate priority 

4 = high priority 

5 = very high priority 



As part of the computation process to be discussed later in Step 4, 
ratings are averaged for each statement and then are graphically 
displayed to demonstrate the relative priority or importance between 
and among the program components. 

The fourth task is to schedule generation, structuring, and map 
interpreting sessions. Typically, one day is needed for the 
generation and structuring steps and a half day is needed for the map 
interpretation step at a later date. 



Step 2 - Generation of Ideas 

Participants The facilitator, using small group processes, often brainstorming, 

generate ideas reminds participants of the focus statement and instructs them to 
relative to a generate ideas relative to the focus. When using brainstorming, 

focus statement basic brainstorming rules are followed. No censoring of other 
people's ideas is allowed; every idea must address the brainstorming 
focus; and routine as well as novel ideas are included. 

When the group is finished generating ideas, mutual understanding 
and consensus on the meaning of each statement is reached and 
redundant statements are deleted These statements are then readied 
for Step 3 - structuring of statements; they are printed on rating 
forms and on index cards. 

As discussed previously, there are ways other than brainstorming to 
generate ideas to represent a conceptual domain. For example, a set 
of statements can be obtained from extant documents such as 
approved program proposals, annual reports, evaluation reports, 
case studies, interviews, or other types of written documents. 
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Step 3 - Structuring of Ideas 



Ideas are 
structured by 
involving each 
participant in 
rating and 
sorting the 
statements 



Structuring ideas involves each participant in two activities that make 
sense of how the statements are related to each other. The first 
activity is rating the statements. Once a set of statements on a given 
focus is generated, participant stakeholders are typically instructed to 
rate each statement on a dimension of interest such as priority or 
importance. The dimension of interest is defined by the rating focus 
described in Step 1. 

The second activity employs a card sorting procedure (Weller & 
Romney, 1990) to obtain information about how the statements are 
related to each other. Individual participants, using a deck of cards 
with brainstormcd statements listed on them, are asked to sort the 
statements into groups or piles that make sense to them. When 
sorting is completed, participants label each of the piles and then 
record the name of each pile on a separate sheet of paper. The 
identification number of each statement belonging to each pile is also 
recorded. 



Sample Sorting Instructions 



1 . Read through the set of cards and sort them in:o piles that make 
sense to you. 

2. Place similar statements together into the same pile. Group for 
similarity, not priority. 

3 . Create as many piles as you want but don't limit yourself to one 
pile and don't have as many piles as there are number of 
statements. 



4 . There are no wrong or right piles. Each person may find that 
the statements can be piled or sorted in several ways. Choose 
the way that seems best to you. 

5 . Provide a label for each of your piles. 

6 . Record each pile on a sheet of paper and indicate by number the 
items belonging to each pile. 
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Step 4 - Computation of Maps 



The Concept 
System is an 
effective means 
of producing 
concept maps 



A general purpose statistical software package which has routines 
for multidimensional scaling and cluster analysis and a graphics 
program to plot the final maps can be used to analyze and compute 
maps* Trochim has written a computer program, The Concept 
System (1989a), for computing maps. 

The Concept System averages the rating data and analyzes the 
similarity sort data. Analysis of data is performed by aggregating 
similarity sort data (Le M higher values in cells indicate a higher level 
of agreement among participants) into a nonmetric binomial 
matrix, then maps are computed through multivariate analysis 
(multidimensional scaling [MDS] and cluster analysis) on the 
nonmetric binomial matrix. See Kruskal and Wish (1978) for 
detailed discussion on MDS and Aldenderfer and Blashfield (1984) 
for discussion on cluster analysis. Finally, average ratings for 
statements and clusters are computed. 



Step 5 - Interpretation of Maps 



A facilitator 
guides 
participants 
through the 
interpretation to 
build 

understanding 
and consensus 
about the final 
map 



Map interpretation includes several steps or activities, all of which 
involve the participant stakehplders. The facilitator prepares the 
appropriate materials and maps and then guides the participants 
through each of the steps. Activities include 

1 . locating the statements on the map 

2 . deciding on the number of clusters 

3. describing the clusters by size, cohesiveness, location, level of 
importance to rating scale values 

4 . naming the clusters 

5 . viewing the map by regions (i.e., cluster groupings) 

6. viewing the map as a whole 

Each of these activities differ in length and complexity, and because 
of group dynamics, the procedures for conducting each step or 
activity varies. It is important to note that this step requires building 
understanding about and consensus regarding the final form of the 
maps. Where disagreement is likely, it is important to ensure 
adequate time for tL i phase of concept mapping* Multiple sessions 
may need to be scheduled to give people time to work through their 
understanding of a map. The point is that where dissent can be 
anticipated, participants must not be rushed. 
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Step 6 - Utilization of Maps 

As was discussed previously, multiple reasons exist for using 
concept mapping. Three are to 

• identify and describe program elements 

• develop program theory 

• plan, implement, and evaluate programs 

Of course, an important part of evaluation is the identification of 
outcomes and that is the use described next in this chapter. 



A Case Study in Using Concept Mapping to 
Identify Outcomes for Vocational 
Education 



The tenets and 
key components 
of vocational 
education can he 
identified 
through concept 
mapping 



The National Center for Research in Vocational Education (NCRVE) 
project staff at the University of Illinois conducted a study to 
systematically identify and examine the basic tenets and key 
components of vocational education as espoused and practiced by 
faculty, administrators, students, and business and community 
leaders at three postsecondary educational institutions: 

• Black Hawk College in Moline, Illinois 

• Dunwoody Institute in Minneapolis, Minnesota 

• Western Wisconsin Technical College in Lacrosse, Wisconsin 

At each site, the tenets and key components of vocational education 
were identified Then, they were displayed in the form of a map, 
which was used as a framework to identify vocational education 
outcomes. To illustrate how concept mapping can be used to 
conceptualize vocational education outcomes, this discussion 
focuses specifically on the concept mapping activities conducted 
with Black Hawk College in Illinois. 



Black Hawk College 

Black Hawk College is a comprehensive public institution located in 
western Illinois on two campuses approximately thirty-five miles 
apart. The largest of the campuses is housed in Moline. The 
combined student enrollment at the two campuses is about 7,200 
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students (Illinois Community College Board, 1992)- Sixty-two 
percent of the student body is female; the same percentage is 
enrolled part-time. About half the students are enrolled in 
prebaccalauieate/transfer programs. Approximately ninety percent 
are white, non-hispanic; about five percent are African-American; 
and five percent are Hispanic. Although combined enrollment is 
roughly ten percent lower than it was in 1981, it is almost thirteen 
percent above the fell 1985 headcount The college employs a total 
of twenty-three full-time administrators and 220 full-time faculty. In 
fiscal year 1991, annual tuition and fee charges were $1,245, 
placing Black Hawk College on the high end of tuition costs in the 
state of Illinois. 



The nature of 
the study and 
purpose of 
concept 
mapping were 
explained 
during initial 
planning 
activities 



Staff from the NCRVE project made an initial series of contacts with 
administrators and faculty from Black Hawk College and conducted 
a plenary meeting in June of 1992. During these contacts and the 
meeting, the nature of this study and the purpose of the concept 
mapping process were explained Specifically, planning was carried 
out -.0 accomplish the following: 

Select vocational education stakeholders for participation in this 
study, 

Develop a focus statement for brainstorming ideas, 
Create a scale for rating ideas, 

Set dates for conducting sessions to brainstorm and structure 
ideas, and 

Finalize a date for the map interpretation session. 



Purpose of this Concept Mapping Project 



Determining the 
basic tenets and 
key Components 
of vocational 
education to 
conceptualize 
outcomes was 
the purpose of 
this study 



The purpose of this concept mapping project was to describe basic 
tenets and key components (i.e., theoretical underpinnings — what 
people say and/or what people do) of vocational education from the 
perspective of Black Hawk College faculty, administrators, 
students, and local business/labor/community constituents. Once 
these basic tenets and key components of vocational education were 
identified and their interrelationships and relative importance known, 
they were visually displayed in map form. The concept map was 
then systematically used by the participant stakeholders as a 
common framework to identify potential vocational education 
outcomes. 
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Participants 



Validity of the brainstormed statements and the capacity of the 
statements to conceptually represent what vocational education ought 
to be are fundamental to the concept mapping process. Construct 
validity was established by the actual participation of stakeholders 
who 

(a) teach vocational education at Black Hawk College, 

(b) employ vocational education graduates in their businesses or 
community agencies, 

(c) administer vocational education, and 

(d) study vocational education. 

Altogether, there were thirty-eight participant stakeholders. Of 
these, twenty-four were male and ten were female. Thirty-five were 
white/Caucasian, two were Native American, and one was 
Hispanic. All die participants were considered to be expert 
representatives for their particular stakeholder group and 
experienced in some aspect of vocational education. 

Thirteen participants were faculty at Black Hawk College, one was 
on the faculty at Western Illinois University, a nearby state-funded 
four-year university, and one was on the faculty at United Township 
High School. These fifteen participants taught in a wide range of 
academic and vocational areas. Many of these individuals also had 
dual or multiple responsibilities other than teaching, such as 
coordinating various types of partnerships with Black Hawk 
College, local businesses, and community agencies; designing 
curriculum and courses relating to vocational education; and 
advising students. 

There were ten administrators. They were employed as deans, vice 
presidents, managers, coordinators or directors of various academic, 
technological, or vocational departments/divisions at Black Hawk 
College. One administrator was employed by Western Illinois 
University. 

Eight participants were from local businesses, labor organizations, 
or community groups. These individuals represented Quad Cities 
Electrical, Graphics Communicators International Union, John 
Deere Harvester Works, Rock Island Arsenal Manufacturing, 
Private Industry Council, and the Department of Rehabilitation 
Services. 

There were five students. These individuals were currently enrolled 
in mechanical technology design, small business administration, 
computer graphics and drafting, or liberal studies at Black Hawk 
College. 
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Brainstorming 



The focus 
statement asked 
participants 
"What 
vocational 
education ought 
to be" 



The focus statement for generating basic ideas regarding the 
conceptualization of vocational education was "Generate statements 
(short phrases and/or words) which describe what vocational 
education ought to be," When the focus statement was used in the 
concept mapping session, participants were told to think broadly 
about the concept of vocational education. They were told that this 
could entail substituting or using collectively such terms as career 
education, occupational education, or technical education. In 
addition, probes were used to stimulate the group's thinking once 
they had begun brainstorming. These probes included 

• What works now (in regard to any aspect of your vocational 
education program)? 

What are the features of your most successful program? 

• What represents best practices? 

• What will the future require? 

The focus statement and probes were intended to elicit thoughts and 
ideas from Black Hawk College participant stakeholders, who 
described what vocational education ought to be. During this 
brainstorming session, over two-hundred and fifty ideas or 
statements were generated. These were culled, to rid them of 
redundancy, into ninety-eight statements (Table 1). 



Table 1 

Brainstortned Statements from Black Hawk 



1 . Be hands on/experimental 

2. Provide incentives to keep students in school 

3 . Teach computer/technology literacy 

4 . Encourage ethical practices/professional 
ethics/work ethic 

5 . Foster a real acceptance of vocational 
education by higher education 

6. Supported by the community (financial* input, 
advisory committees, etc.) 

7. Have relevant Illinois college entrance 
requirements 

8. Provide global perspective/economy 

9. Have a historical perspective 

10. Develop accountability/responsibility for actions 
on the job 



Vocational Education Stakeholders 



11. Articulated between all levels 

12. Provide technical reading/writing skills and teach 
how to follow oral and written directions 

1 3 . Provide leadership development and be a change 
agent 

14. Provide career awareness K-infinity 

15. Encourage nontraditional careers 

16. Ficourage vocational educators to stay current in 
the field, adapt to change (staff development), and 
be involved in action research 

1 7. Prepare individuals with specific 
job/vocational skills 

18. Be enjoyable and personally meaningful, 
interesting, fun 

19. Integrate theory and practice 
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Table 1 (continued) 

Brainstorsned Statements From Black Hawk College Vocational Education Stakeholders 



20. Include reasonable expectations of 
employment (employability/job oriented) 

21. Include cooperation and coordination of secondary 
education, postsecondary education, business, 
industry, labor, and the community 

22. Be proportional in length to the skills being 
taught 

23 . Include basic problem solving 

24. Stress team working/building skills 

25. Teach job seeking skills 

26. Teach students to recognize their skill limits and 
strengths 

27. Stress safety 

28. Provide familiarity with all tools and equipment 
to perform vocational skills, including slang 
terms (business jargon) 

29. Consider environmental awareness 

30. Be useful/practical/valuable for students 

31. Be competency-based with demonstrated 
outcomes 

32. Be accessible (e.g., distance, time, money, 
qualifications, age, etc.) 

33. Be outcomes based 

34. Be a source of pride 

35. Increase student self-esteem 

36. Develop critical thinking 

37. Have curriculum based on work experience 

38. Have on-going assessment and revision 

39. Integrate vocational and academic content 

40. Be adequately funded to ensure quality education 

41. Be multidisciplinary 

42. Encourage creativity 

43. Involve faculty in curriculum development 

44 . Exemplify instructor cooperation and teamwork 

45. Have clear entrance requirements 

46. Have clear exit requirements 



47. Use total quality management to guide practice 

48. Respect/consider ethnic, cultural, and gender 
differences 

49. Teach decision making skills, planning skills, 
forecasting skills, productive thinking skills 

50. Have exemplary facilities for teaching and 
learning 

51. Be accredi t ed between appropriate agencies 

52. Involve professional organizations 

53 . Work with both large and small businesses 

54. Have specific teacher training requirements 

55. Involve state and federal programming (resources, 
networking, etc.) 

56. Apply research findings from vocational 
education 

57. Reflect institutional goals 

58. By physically located with other programs 

59. Provide training for a wide range of jobs/careers 

60. Teach learning how to leam 

61 . Match individual interests, aptitudes, and skills 
with vocations 

62. Be relevant to current and future job markets 
(industry based) 

63. Explore/provide career opportunities and hands-on 
internships 

64. Teach organizational (workplace) and job 
keeping skills 

65 . Rename vocational education to applied education 

66. Provide student and family support services (e.g., 
counseling, advising) 

67. Be taught by an integrated team (system) 

68. Be pre-professionalAechnical level (not 
postgraduate) 

69. Include job shadowing, placement, and follow-up 

70. Provide continual training for employees 
supported by employers 

71. Include health and wellness 
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Table 1 (continued) 

Brainstormed Statements From Black Hawk College Vocational Education Stakeholders 



72. Include people of all ages curriculum design 

73. Include basic skills (e.g., reading, writing, math, 
science) 

74. Prepare students for job requirements/be relevant 
and responsive to meeting industry needs 

75. Teach interpersonal and communication skills 

76. Encourage continuing or lifelong learning 

77. Have active advisory committees involving 
industry and educators 

78. Have state of the art equipment 

79. Overcome negative attitude of vocational training 
by community, business, and higher education 

80. Offer internship programs/on the job training 

8 1 . Employ quality/enthusiastic instructors who have 
practical experience 

82. Use a variety of teaching styles 

83. Help older or new students to adjust to school or 
job environments 

84 . Provide for life experience proficiency/credit 

85 . Be clearly defined, career (path) oriented 



86. Offer comprehensive curriculum 

87. Cover a variety of service areas (different fields) 

88. Offer classes at variable times (flexibility 
scheduling) 

89. Keep up to date with industry/technological 
changes 

90. ' Provide more training in primary and secondary 

programs 

9 1 . Interact directly with business/industry (e.g., 
planned business tours) 

92. Match training with local job market 

93. Provide more self-employment training 

94. Require appropriate prerequisite courses 

95. Ensure that students have access to 
funding/financial aid 

96. Have less administrative interference and more 
decisions by professors and students 

97. Offer students a course on services for people 
with disabilities 

98. Be realistic 



A broad range of 
statements was 
generated by 
brainstorming 



Reflecting the diversity of the participants, a broad range of 
statements was generated. For example, some statements related to 
curriculum and foundational skills: 



be outcomes based 
be hands on/experiential 
be multidisciplinary 
teach computer/technology literacy 
teach basic problem solving 
teach job seeking skills 

Some statements related to the overall ethos of vocational education 
and what it means for its students: 



be a source of pride 
increase student self esteem 

be accessible (i.e., consider distance, time, costs, disabilities) 
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Other statements related to teaching methodology and the credibility 
of vocational education: 

♦ be taught by an integrated team (system) 

♦ use a variety of teaching styles 

♦ be supported by the community (i.e., input, advisory, financial) 

♦ be accredited by appropriate agencies 

Still other statements related to funding and accountability: 

be adequately funded to ensure quality education 

♦ have specific teacher training requirements 

♦ encourage vocational educators to stay current in the field 

The list of ninety-eight statements represents the thinking of the 
participant stakeholders. Their statements specifically described 
what they believed vocational education ought to be. 



Rating of Statements 



A Likert-type 
rating scale was 
used to provide a 
sense of relative 
importance for 
the brainstormed 
statements 



In order to obtain a sense of the relative importance or priority of the 
brainstormed statements, each participant was asked to rate each 
statement on a Likert-type priority scale. The rating instructions 
were rate each statement on a 1 to 5 priority scale where 

1 = very low priority 

2 = low priority 

3 = moderate priority 

4 = high priority 

5 = very high priority 

The ratings were then aggregated and averaged. This rating activity 
was completed by participants relatively quickly, in approximately 
fifteen minutes, on average. 



Sorting 



Cards were 
grouped and 
labeled in ways 
that made sense 
to each 
participant 



The card sorting activity was conducted to obtain each individual's 
thinking on the conceptual structure (i.e., how the brainstormed 
statements were organized and interrelated) of all ninety-eight 
statements. Each participant was given a deck of cards; each card 
listing one of the ninety-eight statements. The participants were then 
instructed to group the statements into piles of similar statements in a 
way that made sense to them. More time was required for this 
activity, ranging from one to one and one-half hours for each 
participant 
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Computation of Maps 



The sort data were entered into Trochim's computer program The 
Concept System (1989b) and were subjected to multidimensional 
scaling (MDS) and then hierarchical cluster analysis. The results of 
the multidimensional scaling analysis were used to make a number 
point map representing all ninety-eight brainstormed statements 
relationally as points on a two-dimensional plot (Figure 1). 



Figure 1. Number Point Map for Vocational Education at Black Hawk CoUege 
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Statements were 
plotted on the 
map according 
to their 
proximal 
similarity to 
other statements 



Each statement, designated by numeric characters that act as 
identification numbers, was plotted on the map according to its 
proximal similarity to other statements. Statements that were sorted 
together frequently are closer to each other on the map than 
statements that were not sorted together frequently. For example, in 
Figure 1, statements 60, 36, 49, 23, and 24 are closely grouped 
together and located in the north east side of the map. These 
statements were thought to be similar by the participants and, thus, 
were frequently sorted together. 

The cluster analysis was superimposed on the MDS results. Decisions 
about the number of clusters were based on statistical criteria and on 
ease of interpretation (i.e., understanding the meaning of the 
statements located within the cluster). Participants in the Black Hawk 
College grouped the brainstormed statements into ten clusters (Table 
2). 



82 

10 J 



Interpretation of Maps 



The concept map resulting from the MDS and cluster analyses of the 
sort data is fully displayed in Figure 2. 



Figure 2. Cluster Priority Rating Map of Vocational Education at Black Hawk College 
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Table 2 

Brainstormed Statements and 
College 



Priority Ratings Grouped 



by Cluster for Black Hawk 



Cluster 1 - Integrated and Outcomes Based 

^ 1. Be hands on/experiential (4.57) 

17. Prepare individuals with specific jot/vocational skills (4.39) 

19. Integrate theory and practice (4.44) 

31. Be competency-based with demonstrated outcomes (4.31) 

22. Be proportional in length to the skills being taught (3.44) 

45. Have clear entrance requirements v 3.44) 

39. Integrate academic and vocational content (4.17) 

94. Require appropriate prerequisite courses (3.33) 

33. Be outcomes based (4.00) 

46. Have clear exit requirements ( 3.81) 

Cluster Average = 3.99 
Cluster 2 - Comprehensive Curriculum 

37 . Have curriculum based on work experience (4.06) 

86. Offer comprehensive curriculum (3.47) 
41. Be multidisciplinary (3.36) 

80. Offer internship programs/on the job training (4.1 1) 

87 . Cover a variety of sei vice areas (different fields) (3.28) 

38. Have on-going assessment and revision (3.94) 

62. Be relevant to current and future job markets (industry based) (4.33) 
57. Reflect institutional goals (2.94) 

92 . Match training with local job market (3.75) 

Cluster Average = 3.69 
Cluster 3 - Career Preparation and Objectives 

8 . Provide global perspective/economy (2.69) 

9. Have a historical perspective (2.08) 

74. Prepare students for job requirements/be relevant and responsive to meeting industry needs 
(4.42) 

93, Provide more self-employment training (2.78) 

30. Be usefuVpractical/valuable for students (4.00) 

63. Explore/provide career opportunities and hands-on internships (4.08) 

20. Include reasonable expectations of employment (employability/job oriented) (4.1 1) 

59. Provide training for a wide range of jobs/careers (3.33) 

98. Be realistic (3.39) 

68. Be pre-professional/technical level (not postgraduate) (2.08) 

85 . Be clearly defined, career (path) oriented (3.64) 

47. Use Total Quality Management to guide practice (3.08) 

84. Provide for life experience proficiency/credit (3.17) 

Cluster Average = 3.30 



Note: Priority ratings are averaged for each statement and presented in parentheses. Each statement was 
rated on a 1 to 5 scale where: 1 * very low priority, 2 = low priority, 3 = moderate priority, 4 = high 
priority, and 5 = very high priority. Cluster averages reflect the mean few all statements grouped 
within each of the clusters. 




Cluster 4 - Foundational Skills 



3. Teach computer/technology literacy (4.17) 

27. Stress safely (3.25) 

12. Provide technical reading/writing skills and teach how to follow oral and written directions 
(4.28) 

64. Teach organizational (workplace) and job keeping skills (3.50) 

28. Provide familiarity with all tools and equipment to perform vocational skills, including slang 
terms (business jargon) (3.53) 

29. Consider environmental awareness (2.89) 

4. Encourage ethical practices/professional ethics/work ethic (3.69) 
36. Develop critical thinking (3.92) 

60. Teach learning how to learn (3.64) 

23. Include basic problem solving (4.00) 

49. Teach decision making skills, planning skills, forecasting skills, productive thinking skills 
(4.28) 

73. Include basic skills (e.g., reading, writing, math, science) (4.08) 

24. Stress team working/building skills (3.86) 

75. Teach interpersonal and communication skills (3.83) 

Cluster average » 3.78 
Cluster 5 - Personal Development 

10. Develop accountability/responsibility for actions on the job (3.78) 

13. Provide leadership development and be a change agent (3.14) 
26. Teach students to recognize their skill limits and strengths (325) 

25. Teach job seeking skills (3.03) 
42. Encourage creativity (3.39) 

7 1 . Include health and wellness (2.58) 

76. Encourage continuing or lifelong learning (3.72) 

Cluster average » 3.27 
Cluster 6 - Student Support Systems 

2. Provide incentives to keep students in school (2.49) 

14. Provide career awareness K-infinity (3.19) 

95. Ensure that students have access to funding/financial aid (3.58) 

34. Be a source of pride (3.78) 

35. Increase student self esteem (3.5) 

6 1 . Match individual interests, aptitudes, and skills with vocations (3.64) 

66. Provide student and family support services (e.g., counseling, advising) (2.69) 

83 . Help older or new students to adjust to school or job environment (3.06) 

Cluster average = 3.24 

Cluster 7 - Diversity and Accessibility 

15. Encourage non-traditional careers (2.97) 

32. Be accessible (e.g., distance, time, money, qualifications, age, etc.) (3.94) 

88. Offer classes at variable times (flexibility, scheduling) (3.53) 

18. Be enjoyable and personally meaningful, interesting, fun (3.86) 

69. Include job shadowing, placement, and follow-up (3.25) 

48. Respect/consider ethnic, cultural, and gender differences (3.58) 

97. Offer students a course on services for people with disabilities (2.40) 

Cluster Average = 3.37 
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Cluster 8 - Faculty/Teaching 



16. Encourage vocational educators to stay current in the field, adapt to change (staff development), 
and be involved in action research (4.47) 

54. Have specific teacher training requirements (333) 

8 1 . Employ quality/enthusiastic instructors who have practical experience (4.42) 
44. Exemplify instructor cooperation and teamwork (3.42) 

67 . Be taught by an integrated team (system) (2,94) 

82. Use a variety of teaching styles (3.72) 

90. Provide more training in primary and secondary programs (3.03) 

Cluster Average = 3.62 
Cluster 9 - External Coordination and Partnerships 

5 . Foster a real acceptance of vocational education by higher education (3.53) 

50. Have exemplary facilities for teaching and learning (3.86) 
65. Rename vocational education to applied education (2.17) 

6. Supported by the community (financial input, advisory committees, etc.) (4.17) 

55. Involve state and federal programming (resources, networking, etc.) (3.08) 
40. Be adequately funded to ensure quality education (4.42) 

70. Provide continual training for employees supported by employers (3.75) 

79. Overcome negative attitude of vocational training by community, business (3.44) 

2 1 . Include cooperation and coordination of secondary education, postsecondary education, business, 
industry, labor, and the community (4.17) 

52. Involve professional organizations (3.69) 

77. Have active advisory committees involving industry and educators (3.92) 

53. Work with both large and small businesses (3.67) 

9 1 . Interact directly with business/industry (e.g., planned business tours) (3.64) 

Cluster Average = 3.65 
Cluster 10 - Administrative Support 

7 . Have relevant Illinois college entrance requirements (2.97) 

96. Have less administrative interference and more decisions by professors and students (2.78) 
43. Involve faculty in curriculum development (4.1 1) 

51. Be accredited between appropriate agen (3.94) 
1 1 . Articulated between all levels (3.56) 

58. By physically located with other programs (2.31) 

56. Apply research findings from vocational education (3.22) ' 
72. Include people of all ages in curriculum design (2.47) 

89 . Keep up to date with industry/technological changes (4.53) 

78 . Have state of the art equipment (3.83) 

Cluster Average = 3.37 



During the three-hour interpretation session, participant stakeholders 
examined the statements in each cluster for mutual understanding 
and consensually named each of the ten clusters. The participants 
then examined the named clusters and, aided by the statistical 
analysis produced by The Concept System, further grouped these 
clusters into five regions: foundations, curriculum, administration 
and partnerships, faculty, and support services (Table 3). 
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Table 3 

Listing of regions and clusters for Black Hawk College 



Region 


Clusters 


I: Curriculum 


1. Integrated and outcome based 

2. Comprehensive curriculum 

3. Career preparation and objectives 


II: Foundations 


4. Foundational skills 

5. Personal development 


HI: Support Services 


6. Student support systems 

7. Diversity and accessibility 


IV: Faculty 


8. Faculty/teaching 


V: Administration and 
Partnerships 


9. External coordination and partnerships 
10. Administrative support 



These five regions and their associated ten clusters are easily 
discerned in Figure 2, previously displayed Beginning in the east 
and moving counter clockwise, three clusters — named integrated 
and outcomes based, comprehensive curriculum, and career 
preparation and objectives — contain statements that describe the 
curriculum region. 

Two clusters, located in the northeast named personal development 
and foundational skills, contain the tenets of vocational education 
that describe the foundations region. 

The third region, support services, lies in the northwest. This 
region contains two clusters: student support systems and diversity 
and accessibility. These clusters contain descriptions of various 
types of student support systems and descriptions of what ought to 
be in terms of (1) understanding the diverse cultural differences of 
vocational education students and (2) making vocational education 
accessible to all potential vocational education students. 

The broad area on the south side of the map contains two clusters, 
external coordination and partnerships and administrative support. 
These clusters describe the administration and partnerships region. 

The west side of the map, with a single cluster, pertains to faculty 
and teaching. This cluster, representing the faculty region, pertains 
to the quality of faculty and the effectiveness of the faculty in 
reaching the vocational education needs of their students. 
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The cluster 
rating map 
shows 

relationships 
within 
vocational 
education and 
tenets that 
describe its key 
components 



In its entirety, the cluster rating map of vocational education 
demonstrates the various relationships within vocational education at 
Black Hawk College and simultaneously shows individual elements 
or basic tenets of what ought to be in vocational education. The map 
also displays clusters of elements or tenets that describe key 
components of vocational education. It also displays regions of 
clusters or key junctional areas of vocational education. 

Figures 1 and 2 reveal three levels of information that describe what 
vocational education ought to be, as perceived by the participant 
stakeholders. The lowest level reveals the basic tenets of vocational 
education by the points representing the various brainstorm 
statements (Figure 1). The next level reveals key components of 
vocational education through the ten clusters (Figure 2) and the 
highest level reveals the broader context of vocational education via 
the regions (Figure 2). 

Figure 2 also depicts these levels of information in relation to each 
other and the whole, and it reveals the relative priority of vocational 
education tenets by cluster and by region. The height of the cluster 
boundary (i.e., fence) indicates die relative priority rating for each 
cluster. The higher the fence bordering each cluster, the higher the 
average rating for all the statements within that cluster, the lower the 
fence, the lower the average rating. 

As is clearly shown by the height of the fences in Figure 2, the 
integrated and outcomes-based cluster has the highest relative 
priority rating, with a priority rating between 4.07 and 4.57 on the 
5.0 scale. The next highest priority rating comes from two of its 
neighboring clusters, foundational skills and comprehensive 
curriculum. These clusters were rated between 3.58 and 4.07. The 
external coordination and partnerships cluster and the 
faculty/teaching cluster were next in priority; they were rated 3.08 to 
3.58. 

The remaining five clusters — student support services, diversity and 
accessibility, personal development, administrative support, and 
career preparation and objectives — had the lowest relative priority 
ratings. They were rated 2.08 to 2.58 in level of priority. See Table 
2 for the actual average priority rating values for each statement and 
for each cluster. 



Vocational Education Outcomes 



Outcomes were 
articulated 
using the map 
as a framework 
to guide 
participants 



After reaching mutual understanding and consensus on the meaning 
of the cluster rating map of vocational education, the participant 
stakeholders of Black Hawk College were asked to articulate 
outcomes of vocational education using the map as a framework to 
guide them. Participant stakeholders were asked to state what the 
vocational education outcomes would be for each of five identified 
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regions if the tenets, as described in the clusters in each of the five 
regions, were carried through. The specific directions provided by 
the facilitator encouraged participants to think in terms of changes 
(e.g M increases or decreases) that could provide a basis for building 
measurements. Outcomes articulated by the participants are revealed 
in Table 4. 



Table 4 

Vocational Education Outcomes for Each Region on the Cluster Rating Map for Black 
Hawk College 



REGION #1 CURRICULUM— Encompasses the following clusters: (1) Integrated and Outcomes Based, (2) 
Comprehensive Curriculum, and (3) Career Preparation and Objectives. 

1 . Increase in number of employable and productive individuals. 

2. Increase integration of vocational/academic skills and programs. 

3. Increase length of employment 

4 . Decrease in customer complaints (complaints of business/industry about graduates). 

5. Decrease in educational barriers. 

6. Increase student enrollment/retention. 

7. Increase student involvement 

8 . Increase in in temship/on-the-jdb opportunities. 

9. Increase in usefulness, practicalness and relevance of courses. 

10. Increase in understanding of exit requirements. 

1 1 . Increase in use of outcomes based education. 

1 2. Increase in involvement of business and industry, students, and other stakeholders in curriculum 
development 

REGION #2 FOUNDATIONS— Encompasses the following clusters: (4) Foundation Skills and (5) 
Personal Development 

1 . Increase in number of students mastering basic skills (reading, writing, math, computer skills). 

2. Increase self-esteem. 

3 . Increase creativity. 

4 . Decrease worker absenteeism. 

5 . Increase worker flexibility within job. 

6. Increase demand for further training. 

7. Increase ability to change job. 

8 . Decrease in job stress. 

9 Increase in teamwork skills. 

1 0. Increase interpersonal/communications skills. 

11. Decrease in job accidents. 

12. Decrease in unethical behavior. 

1 3 . Increase in higher-order cognitive skills. 

14. Decrease in labor cost/increase in efficiency. 

15. Increase in technological literacy. 

16. Decrease in medical/insurance/worker compensation claims. 

1 7 . Decrease in needed on-the-job during initial employment 

REGION #3 SUPPORT SERVICES— Encompasses the following clusters: (6) Student Support Systems 
and (7) Diversity and Accessibility. 

1 . Increase program completion rates. 

2. Improve student self-image. 

3. Increase morale. 
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4 . Improve student/career program match. 

5 . Reduce the "glass ceiling." 

6. Increase in fhtancial aid to students. 

7 . Increase in student support services (whatever they need). 

8. Increase student enrollment 

9 . Increase in services/accessibility for w other"-abled students. 

10. Increase employment for H other"-abled students. 

1 1 . Increase in quality of product 

1 2. Increase of individuals with disabilities in workforce. 

13. Increase in non-traditional students served. 

14. Increase in learning. 

1 5 . Increase in career exploration. 

1 6. Increase in open entry/exit and flexible scheduling. 

REGION #4 FACULTY— Encompasses the Faculty/Teaching cluster (8). 

1 . Increase teacher satisfaction. 

2. Increase teacher pay. 

3 . Increase match of teaching styles with learning styles. 

4 . Increase teacher enthusiasm. 

5 . Increase teachers seeking continuing education/faculty development 

6 . Increase instructors' field experience. 

7 . Increase teacher retention. 

8 . Improve teacher training requirements. 

9 . Increase research activity. 

1 0. Decrease student frustration. 

1 1 . Increase teacher/student interaction. 

1 2. Improve classroom learning environment. 

1 3 . Decrease teacher loneliness. 

14. Decrease student alienation. 

1 5 . Decrease teacher alienation. 

1 6. Increase of utilization of research findings. 

1 7 . Improve evaluation/assessment of teaching/learning process. 

1 8 . Increase in useful research. 

19. Increase in ability to make learning fun. 

REGION #5 ADMINISTRATION AND PARTNERSHIPS— Encompasses the following clusters: (9) 
External Coordination and Partnerships and (10) Administrative Support 

1 . Increase funding to ensure quality education. 

2. Increase cooperation between schools and business. 

3 . Increase continuity. 

4 . Increase employer involvement in program design. 

5 . Improve image/acceptance of vocational education. 

6. Improve design of programs to be articulated between school and work. 

7. Increase in current technology. 

8 . Decrease in administrative barriers. 

9 . Improve articulation from kindergarten through work. 

1 0. Increase positive attitude of business and students. 

1 1 . Increase money spent on computers and technology. 

1 2. Clearer focus on educational priorities through enhanced partnerships. 

1 3 . Improve facilities, equipment, and resources. 

14. Improve college entrance requirements that foster career education. 
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As espoused by the vocational education participant stakeholders, 
these broadly-based outcomes, by inference, are linked directly to 
specific regions of the map (Figure 2). Thus* they are also linked to 
specific named clusters and to specific tenets of vocational 
education located within the clusters. These identified outcomes can 
now provide a clear basis for 

1) setting goals and objectives, 

2) determining ways to measure program and student success, 

3) improving vocational education programs, and 

4) improving institutional effectiveness by understanding vocational 
education and its intended outcomes. 



Conclusion 



Maps can define 
outcomes and 
facilitate making 
informed 
decisions for 
vocational 
education 
evaluation 



The purpose of this concept mapping project was to identify the 
basic tenets and key components of vocational education in order to 
determine vocational education outcomes. The maps implicitly or 
explicitly describe what vocational education experts believe in, 
what they practice. The maps clearly define what vocational 
education ought to be and were easily employed to identify 
outcomes. The maps and accompanying information can be used 
for other purposes. They can facilitate policy makers and program 
planners and managers in making informed decisions in regard to 
setting policy and to planning, implementing, and evaluating 
vocational education. 
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Chapter 6 



Outcomes Assessment in 
Vocational Education 

N. L. McCaslin 



Overview Federal and state governments, businesses, and the public axe 

calling for increased accountability at all levels of education. In 
vocational education, this accountability can be reflected in the 
assessment of educational, economic, and psycho-social outcomes. 
This chapter presents a framework and foundation for the selection 
of outcomes; the identification of indicators; and the procedures for 
collecting, analyzing, and reporting of information. Combined 
with the concept mapping process described in Chapter 5 which 
can be used to identify outcomes, this information provides a basis 
for selection of performance assessment techniques described in 
Chapter 7. 

In this chapter This chapter presents a discussion of the following topics: 
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Relationship of Outcomes Assessment to Other 
Types of Evaluation 



Results of 
student 

achievement are 
the focus of 
reform, but 
outcomes 
assessment can 
fit into a more 
comprehensive 
evaluation 
framework 



Much of the current literature on educational reform could lead one 
to conclude that the only important evaluation information is that 
related to assessing results in terms of student achievement. 
However, if student achievement information is all that is 
available, how can teachers and administrators know what worked 
and did not work? Upon what basis should decisions to add and 
drop programs be made? What should be changed in order for 
vocational students to achieve at higher levels? 

Those concerned with evaluating the outcomes of vocational 
education should consider how outcomes assessment fits into a 
broader and more comprehensive approach to evaluation. This 
approach requires the use of valid and reliable information of three 
major types: 

• the needs for vocational education as expressed by clients (i.e., 
students, employers, and society), 

the educational processes followed by vocational education 
programs, and 

the outcomes achieved by students in vocational education 
programs. 

A more detailed explanation of these types is presented in A 
Framework for Evaluating Vocational Education (McCaslin, 
1990). The interrelationships of these three types of evaluation 
information are presented in Figure 1. 



Figure 1. Interrelationships of evaluation information 
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Vocational Education Needs 



Vocational Individuals choose to participate in postsecondary vocational 

education is education on a voluntary basis* Individuals generally will not 

customer-driven enroll unless they see the opportunity to benefit from participation 

and broadly in the program* Therefore, vocational education can be viewed as 

defined to include a "customer" driven program* "Customers" in this case are broadly 

all groups in the defined to include all groups in a community that might have a 

community stake in vocational education (Rossi & Freeman, 1991)* 

Kaufman and Stone (1983) indicated that in assessing needs it is 
necessary to show and document gaps between current results and 
desired results (in terms of outcomes) and to then place these gaps 
or needs in priority order. Accordingly, needs are concerned with 
determining the extent or magnitude of a problem. Needs can be 
determined by reviewing existing information or collecting new 
information. This information might be of either a quantitative or 
qualitative nature. Needs provide the basis for development and 
implementation of educational processes. 



Vocational Education Processes 



Evaluation of 

vocational 

education 

processes 

provides 

valuable 

information 



Process information provides evaluators with a basis for 
understanding and interpreting what has occurred. Process 
includes information such as program goals and objectives, length 
of course, linkages with business and industry, use of advisory 
councils, type and kind of job placement services provided, and 
safety. Many states' five-year reviews of their vocational education 
programs have relied on this type of information for program 
approval. Program sponsors/funders typically place heavy 
emphasis on process information in demonstrating the 
accountability of their programs. 

If vocational education programs are to be improved as a result of 
an evaluation, it is essential to provide information about how they 
have been conducted. Rossi and Freeman (1991) have referred to 
the evaluation of processes as systematic monitoring to examine 
program coverage and delivery. Of course, conclusions that can be 
drawn about the effectiveness of processes are often based on the 
outcome? that are produced by them. 



Vocational Education Outcomes 



The current emphasis in educational evaluation is on documenting 
educational effectiveness in terms of student achievement. This 
type of effectiveness has been referred to as outcomes, impact 
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Outcomes, 

impact, or 

product 

evaluation 

document 

educational 

effectiveness 



(Rossi & Freeman, 1989), or product (Stufflebeam & Shinkfield, 
1985) evaluation. For the purposes of this chapter, the term 
outcomes evaluation will be used 

Outcomes in postsecondary vocational education refer to the 
results or accomplishments of those who participate in these 
programs (McCaslin, 1990). These outcomes should be easily 
understood and agreed upon by vocational education stakeholders. 
Among the questions that outcomes evaluation can answer are the 
following: 

• What results were obtained? 



Single or 

composite 

outcome 

measures 

should be 

used 



• How well were needs reduced? 

• What should be done with the program after it has run its 
course? 

An outcome might consist of either a single statistic or a composite 
measure based on several factors. The terms core standards, 
measures of performance, and performance standards are 
sometimes used in reference to educational outcomes (e.g., the Carl 
D. Perkins Vocational and Applied Technology Education Act of 
1990 and the Job Training Partnership Act). 



Types of Vocational Education Outcomes 



A wider range of 
outcomes for 
vocational 
education are 
currently 
purported 



The outcomes of vocational education often have been thought of 
solely in economic terms. However, more recent discussions and 
studies have included a wider range of educational, psychological, 
and sociological outcomes. In this chapter, three major types of 
outcomes are described: economic, educational, and psycho-social. 



Economic Outcomes 



Economic outcomes are usually measured in terms of the 
experiences former vocational education students have in the labor 
market. Examples of economic outcomes include the following: 
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Examples of Economic Outcomes 



• Labor force participation rates 

• Employment and unemployment rates 

• Training-related placement 

• Type of employment 

• Earnings 

• Employee satisfaction with work 

• Employer satisfaction with employees 



Labor force participation rates refer to the extent to which 
individuals are actually working or seeking work. It does not 
include those individuals who have given up and stopped seeking 
work even though they are unemployed 

The outcomes of employment and unemployment rates are 
determined by the percentage of individuals who are either 
working or not working. This outcome as well as the previous one 
(labor force participation rates) can be compared with state and 
national statistics for similar groups of individuals if the same 
definitions have been used in computing the rates. 

Historically, training-related placement has been the major 
outcome for evaluating vocational education. This outcome refers 
to the percent of individuals who find employment in the 
occupations for which they were trained. This outcome has been 
criticized since its achievement can be determined by factors that 
are beyond the control of vocational educators. 

The type of employment has also become an important outcome 
as emphasis on vocational education's role in economic 
development has increased. Since many pew jobs are in small 
businesses, self-employment has become an important outcome for 
vocational education and many entrepreneurship programs have 
been developed with this specific outcome in mind. The percentage 
of males and females distributed across all types of occupations is 
another important factor and many programs have been specifically 
designed to address inequities in these occupational areas. 

Earnings represent another type of outcome for vocational 
education programs. In determining earnings, it is usually helpful 
to think in terms of annual earnings that reflect both hourly wages 
and number of hours worked. Annual earnings also allow 
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cvaluators to compare the wages of those who work as an 
employee with those who are self-employed Also, it is important 
to examine and compart the earnings of those who are employed in 
the occupations for which they were trained with those who are not 
working in those occupations. 

The degree of employee satisfaction with work is classified here 
as an economic outcome since it is often reflected in the 
productivity of an individual. Although employee satisfaction is 
influenced by many factors beyond the influence of vocational 
education, it can be used to examine how the student's program 
developed a realistic picture of the work place. 

Similarly, employer satisfaction with employees is classified as an 
economic outcome since it can be a reflection of a businesses' 
success in the marketplace. This outcome also can be influenced 
by many factors beyond employer satisfaction with individual 
employees. Therefore, evaluators need to be cautious in 
interpreting the results. 



Educational Outcomes 



A move 

toward 

instructional 

outcomes is 

advocated by 

legislators 

and 

policymakers 



Legislators and policymakers are moving more and more toward 
accountability measures tied to the instructional outcomes of 
educational agencies. This reflects a departure from the 
institutional or process outcomes used in the past This growing 
concern for educational outcomes is being reflected in revisions to 
state-wide approaches to vocational education evaluation in states 
such as Illinois, Minnesota, and Virginia. Possible educational 
outcomes follow: 



Examples of Educational Outcomes 

• Basic educational skills 
Higher-order thinking skills 
Knowledge of the world of work 
Occupational skills 
Attendance and dropout rates 
Continuing education rates 
Students satisfaction with education 
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The development of basic educational skills through courses in 
mathematics and English are perceived by more than eighty-six 
percent of the American public as important for all students (Elam, 
1980). The high level of importance placed on these subjects 
indicate that vocational educators should emphasize the 
development of these skills in their courses by showing students 
how they can be applied. 

Higher-order thinking skills require students to do more than 
simply recall or understand information. The development of these 
higher-order thinking skills require students to apply, analyze, 
synthesize, and/or evaluate information (Bloom, 1956). If students 
are to develop these types of skills, the objectives, instructional 
activities, and evaluation processes need to reflect this emphasis. 

Another educational outcome is knowledge of the world of work. 
If, as Cetron and Davies (1989) predict for the next decade, new 
jobs will appear and obsolete jobs will be gone faster than ever 
before — individuals will need to know the type of jobs that will be 
available to them as the work force changes. 

Occupational skills have been the major outcome of interest to 
many postsecondary vocational educators. Much of the evidence 
for this outcome has been provided through the use of competency 
and/or performance testing. 

The levels of school attendance and dropout rates are 
particularly important outcomes. If our country is to increase the 
educational level of students, it is imperative that attendance rates 
be increased and dropout rates be decreased. These rates should be 
examined for differences among races, given the wide disparity in 
the rates for these groups. 

The continuing education rate for students represents an outcome 
for vocational education measured only recendy. Laughlin (1986), 
in a national longitudinal study of more than 6000 individuals, 
reported that sixty-two percent of vocational education students 
initially enrolled in some type of postsecondary education and 
training. Of those who enrolled, sixty-two percent completed or 
were still enrolled at the time of the follow-up. 



Psycho-Social Outcomes 



Psycho-social 
outcomes relate 
to the personal 
development of 
students 



The psycho- social outcomes of vocational education relate 
primarily to the personal development of the student. These 
outcomes are often associated with participation in vocational 
student organizations. Some of these outcomes are difficult to 
measure and indicators of their effectiveness are not readily 
available. A few of these outcomes follow: 
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Examples of Psycho-Social Outcomes 



• Aspirations 

• Attitudes and values 

• Self-esteem 

• Citizenship 

• Leadership 



The aspirations of students refer to their desires and ambitions. 
Experiential learning provided by vocational education provides 
students with real-life examples to use in planning their future. In 
many cases personal contacts with individuals in the workplace can 
provide the opportunity for students to identify role models that 
they wish to emulate. 

Employers often criticize the attitudes and values that individuals 
bring to the work place. Teachers and administrators have the 
opportunity to influence students' attitudes and values by the 
personal examples they give to students. This outcome can also be 
influenced through the participation of a student in a work-based 
learning program. 

Self-esteem refers to the belief students have in themselves. It also 
refers to their self respect. This outcome can be influenced by how 
other individuals and faculty interact with students. Positive 
learning environments need to be established by teachers, 
administrators, job supervisors, and other students and co-workers. 

The outcome of citizenship refers to how students demonstrate 
their membership in society by acts such as voting and 
participating in community organizations. 

Leadership skills will be needed throughout a student's life. These 
skills can be developed by providing students with opportunities to 
direct and guide their own learning experiences in vocational 
education. 



Assessing Vocational Education Outcomes 



Assessing outcomes of vocational education, like any other 
evaluation, requires a carefully thought out plan. A worksheet (see 
Figure 2) for planning outcome assessments can be used to assist 
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Outcomes 

assessment 

requires 

careful 

planning 



with these efforts- Five steps need to be taken in completing the 
worksheet: 

1 . Select outcomes to be assessed, 

2. Identify indicators for each outcome, 

3. Identify information collection arrangements, 
4* Determine the analysis procedures, and 

5, Report the information. 



To begin the 
process, select 
outcomes 



Select Outcomes To Be Assessed 

The first step in completing the worksheet is to select the outcomes 
that are to be assessed The economic, educational, and psycho- 
social outcomes discussed earlier in this chapter provide an initial 
listing for most postsecondary vocational education programs. 

The Carl D. Perkins Vocational Applied Technology Education 
Act of 1990 requires all vocational education programs to evaluate 
their programs using the following outcomes: 

L measures of learning and competency gains, including student 
progress in the achievement of basic and more advanced 
academic skills and 

2. one or more measures of performance, which shall include 
only 

♦ competency attainment; 

♦ job or work skill attainment or enhancement, including 
student progress in achieving occupational skills 
necessary to obtain employment in the field for which 
the student has been prepared, skills which include 
occupational skills of the industry the student is 
preparing to enter 

♦ retention in school or completion of secondary school 
or its equivalent; and 

♦ placement into additional training or education, 
military service, or employment (American Vocational 
Association (AVA), 1990, p. 70). 

Often state educational agencies have additional outcomes for 
which they want evaluation information. Local postsecondary 
agencies may also require information regarding even other 
specific outcomes. All of these factors need to be considered when 
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selecting the outcomes to be assessed. The outcomes selected for 
assessment should be listed in the first column of the worksheet 



Identify Indicators for Each Outcome 

Second, potential indicators need to be identified for each outcome 
that will be assessed (Oakes, 1986). These indicators can be 
quantitative or qualitative in nature. Many vocational programs 
The second step have well-established performance testing programs to determine 
is to identify the occupational skill of their students. For example, the National 

indicators Occupational Competency Tests could be used as an indicator. 

Other programs may wish to select academic skills as an outcome 
of interest and use tests to determine the mathematics, reading, or 
writing ability of their students. When relevant indicators are 
available, the assessment process is simplified greatly. 

In other cases, new sources of data and information will need to be 
developed as indicators. The emphasis on articulation has resulted 
in many programs where students can get advanced standing in 
four-year coUeges and universities (e.g., 2+2+2). In this case a 
postsecondary institution might have selected the continuing 
education rate of its students as an outcome to be assessed. Since it 
is a new area, information regarding this outcome may not be 
available and a new effort may be needed to gather this 
information. 



Identify Information Collection Arrangements 



The third step 
is to designate 
data 

collection 
arrangements 



The third step in assessing outcomes is to designate the 
arrangements for collecting the information needed. Details are 
needed such as 

• Who will collect the data or information? 

What conditions will be needed to collect data? 
When will data collection need to be completed? 

These questions are of paramount importance since most 
postsecondary institutions will want to assess their outcomes as 
objectively as possible. 

Identifying the person who will collect the information is an 
important decision to be made in planning how to assess outcomes. 
If the outcomes are being assessed to provide summative 
information for accountability purposes, it would be advisable 
to select an individual who is external to the postsecondary 
institution and who would be viewed as more objective and 
credible. On the other hand, if the outcomes are being assessed 
primarily to provide formative information for program 
improvement then a person internal to the postsecondary institution 
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might be identified because of his or her competence in evaluation 
and knowledge of the existing programs. 

Another important task in identifying conditions for collecting 
information is to specify how the information will be collected. 
This specification should answer questions such as 

♦ If existing information is to be used, will permission of 
students be required to obtain this information? 

♦ If information is to be compared longitudinally, how will this 
information be retained? 

• Will a sample or a population of students be used to collect the 
information? 

• Will information be collected directly from the students or will 
teachers, administrators, or counselors be used to collect the 
information? 

In completing the section of the worksheet, it is imperative to 
indicate when information will be collected Often the time is 
determined by an outside agency. For example, the state 
department of education or board of regents may require certain 
types of outcome information by a specified date in order for a 
postsecoi dary institution to receive funding. Also, time frames 
could be specified by the requirements of management information 
systems or other accountability mechanisms established by the 
postsecondary institution. 



Determine the Analysis Procedures 



The fourth 
step requires 
determining 
analysis 
procedures 



The fourth step in completing the worksheet is to determine the 
analysis procedures. The major purpose of the data analysis 
process is to reduce and synthesize the information into a more 
manageable and informative set of data. In assessing outcomes, 
two basic types of indicators were specified earlier — quantitative 
or qualitative. 

Much of the analysis required for assessing outcomes that use 
quantitative data can be analyzed using descriptive statistics. 
Qualitative data, on the other hand, can be analyzed using content 
analysis, key incidents, analysis in the field, and searching for 
patterns and categories* Both of these types of analysis require 
careful and systematic study and interpretation. 



Report the Information 

The last step is to The final step in completing the worksheet is to decide how the 
report results to information will be reported. The key questions to be addressed 
stakeholders include: 
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To whom is the report intended? 



How will the information be reported? 



Findings should 
be presented in a 
timely manner to 
have the 

maximum impact 
on postsecondary 
vocational 
education 



• When will the report be available? 

Each of these questions will be described briefly in this section. 

Identification of the intended audiences for information regarding 
the outcomes of a postsecondary vocational education program is 
an essential ingredient in the assessment process. One of the most 
important audiences is local program stakeholders. These 
stakeholders include faculty, administrators, counselors, 
employers, and students. These are the people most directly 
affected by a program and the ones who need information to use in 
improving them. 

Another important audience is the State Department of Education 
or the Board of Regents. These agencies often provide funding and 
are looking for information regarding the effects of this funding. 

The U. S. Department of Education is another an audience for the 
report. Within the Department of Education, the Office of 
Vocational and Adult Education has primary responsibility for 
administering the federal vocational education legislation. As 
mentioned earlier, the Carl D. Perkins Vocational and Applied 
Technology Education Act of 1990 specifies the need for 
information regarding the effectiveness of vocational education. 

Specification of how the information will be reported is yet another 
important task. One of the most common formats is a printed 
report. However, there are numerous other ways to report 
evaluative information such as oral reports, video-tapes, and 
multimedia presentations. Still other formats include fact sheets 
with questions and answers, executive summaries, public 
testimony, and debates. No matter what format is selected it is 
important to keep the message simple, straight forward and easy to 
understand. 

Finally, it is important to indicate when the report will be available. 
To make the assessment of outcomes most valuable, results must 
be presented in a timely manner for relevant discussions and 
decisions. In some instances timeliness for submitting reports are 
specified by sponsors well in advance. Each individual responsible 
for assessing outcomes needs to time the release of reports to have 
the maximum impact on improving programs that are offered by 
postsecondary institutions. 
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Chapter 7 

Performance Assessment 

Linda Mabry 



Background 



Problems associated with standardized educational testing have 
become increasingly well-documented (see Berlak, 1992a; Cannell, 
1987; Cooley&Bernauer, 1991; Haladyna, Nolen, & Haas, 1991; 
Linn, 1991; Massey, 1989; Smith & Rottenberg, 1991; Wilson & 
Corbett, 1989). At the same time, interest in performance 
assessment as an alternative means of representing and assessing 
learners' competence has grown. Performance assessment is now 
being explored across the country at every educational level — 
elementary, secondary, and postsecondary. 

Vocational educators have long used performance assessment to 
evaluate student learning (Finch & Crunkilton, 1984; Wentling, 
1980). Today, they are reexamining traditional notions of 
performance assessment, seeking to expand and improve their use 
of these methodologies. While experience with performance 
assessment is relatively great in the field of vocational education, the 
need for these evaluation techniques may be relatively great, as well. 
There is opportunity for reciprocation: The field can learn from and 
build upon vocational education's experience with performance 
assessment while vocational educators can learn from the developing 
field and adapt new strategies to their particular needs. 



In this chapter The plan of this chapter is to introduce the topics shown below: 
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Performance Assessment Techniques 



The terms "performance assessment," "alternative assessment," 
"authentic assessment," and "direct assessment" are common in 
current discussions of measuring student outcomes. While their 
meanings greatly overlap, each of these terms suggests a different 
aspect of controversial "new" assessment methods. 



Type Purpose 



Performance Signals that something students 

assessment actually do is being evaluated 

Alternative Signals that evaluation is based on 

assessment something other than standardized 



As are the others, performance assessment is an umbrella term for a 
number of techniques for evaluating what students know and can 
do. Each of these techniques, in turn, can be adapted by 
practitioners to respond to various contexts, purposes, and levels of 
sophistication. What follows is a nonexhaustive overview of 
performance assessment techniques suggesting some of the 
variations associated with each. 



Profiles and Records of Achievement 

In Britain and Australia, there is a dazzling variety of profiles and 
records of achievement Some are teacher narratives of student 



testing or similar formats 



Authentic 
assessment 



Signals that assessment tasks 
presented to students elicit 
worthwhile responses reflective of 
real or real-world competence 



Direct 
assessment 



Signals that the evaluation focuses on 
what we want to measure rather than 
a proxy for it. (E.g., in direct writing 
assessment, student writings are 
evaluated rather than responses to 
multiple-choice questions about 
grammar, usage, and vocabulary.) 
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Innovators at 
state, district, 
and local levels 
are discovering 
and 

rediscovering 

implementation 

issues 



performance; they may be structured or they may be unrestricted by 
guidelines of any kind Some axe computer-generated, with teachers 
selecting coded phrases to describe student behaviors and then 
feeding these descriptors into the computer for compilation into a 
report Some resemble expansions of familiar report cards that use 
"grids" of teacher-determined letters or numbers that are implicitly 
norm-referenced evaluations of student performance in prespecified 
categories. 

Some profiles and records are series of continua (from, for example, 
"requires specific direction" to "often initiates independent effort") 
along which teachers rate student performance* Some continua label 
each of a series of stages between the two poles (for example, 
"requires specific direction," "works independently if general goals 
are stated," "sometimes initiates independent effort," "often initiates 
independent effort") implying a sequentially and a consistency in 
gradations of difficulty which may not exist Some continua gather 
evidence of student progress over time; others are summative 
evaluations of final accomplishment (Broadfoot, 1987). 

In the U.S., recent interest in performance assessment has not been 
accompanied by a nation-wide effort to develop assessment 
strategies across a wide range of curricula. For the most part, 
efforts to develop and implement these measures have been isolated 
and independent Consequently, innovators at state, district, and 
local levels are discovering and rediscovering common philosophical 
and implementation issues. Some issues pertinent to the 
development and implementation of profiles are suggested below. 



Profile Implementation Issues 



• To whom does a profile belong, student or college? 

• Who protects confidentiality and determines access to a 
student's profile? 

• Who shall determine what is reported, instructor or student? 

• Is validity compromised by bias if the student ultimately decides 
what a profile includes and excludes or if a instructor does? 

• Shall nonschool people such as employers and parents 
contribute to student profiles? 

• Shall differences of opinion about student accomplishment be 
homogenized by a process of consensus-building, or shall 
variety in judgments be expected and preserved? 
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♦ What shall be the provision for student negotiation or appeal? 

♦ Shall only final products be collected or also drafts which might 
illuminate the student's work habits, response to critique, and 
ability to generate and develop ideas? 

♦ Shall contents be exclusively academic or shall evidence of out- 
of-school accomplishments be collected? 

♦ Shall evidence of cognitive ability dominate; shall attention also 
focus on affective and conative 1 characteristics? 

♦ Shall faculty (and others) be provided guidelines for describing 
student accomplishment so as to minimize neglect of important 
evaluative dimensions; or is free response, which promotes 
unique student descriptions, more beneficial in coming to 
understand a particular student's achievement? 



Cumulative 
resumes can 
communicate 
student 
progress 
toward linking 
learning in 
school to the 
workplace 



Profiles and records of achievement may be particularly informative 
as vocational educators begin to develop the kinds of cumulative 
resumes called for in the Secretary's Commission on Achieving 
Necessary Skills (SCANS) Learning a Living report (1992). 
Cumulative resumes are intended to contain information about 
students, beginning in middle school, in order to document progress 
in courses taken, projects completed, and competence attained. 
According to SCANS, a new voluntary assessment system that 
incorporates new methods of assessing performance can help 
communicate about student performance and certify progress toward 
standards that link what is learned in school to the workplace. 



Portfolios 

Examples of A portfolio may be defined as "a purposeful collection of student 
student work are work that exhibits the student's efforts, progress, and achievements 
included in in one or more areas. The collection must include student 

portfolios to participation in selecting contents, the criteria for selection, the 
promote a broad criteria for judging merit, and evidence of student reflection" 
representation of (Paulsen, Paulsen, & Meyer, 1991, p. 60). 
achievement 

Portfolios are conceptually similar to profiles and records of 
achievement in promoting broad representation of student 
achievement and in preserving opportunity for student input, 



I'The conative components [of ability] are those concerned with determination, persistence, and will . . . 
[often] ignored or inappropriately subsumed under affective. Yet a person can very much enjoy doing 
something without being determined to see it through— and he or she can hate doing something but still be 
determined to do it" (Raven, 1992, p. 218). 
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reflection, and self-evaluation. But whereas profiles typically 
describe student work, portfolios also include examples of it. 

Portfolios differ from two other types of collections of student work 

1 . cumulative folders — formal, summative, largely academic and 
demographic, and teacher-directed to the exclusion of student 
input; and 

2 . student journals — informal, nonevaluative, and reflective, 
without documenting performance. 

In postsecondary education, portfolios have been widely used in 
certain areas including art, photography, and architectural 
technology (Wolanski, 1992). Today, practitioners are 
experimenting with portfolios in almost every field of study. At 
Shoreline Community College in Washington State, portfolios are 
being used to assess student learning in the classroom and readiness 
for promotion within the writing curriculum, and to measure 
program and institutional effectiveness (Hart, Rollman, & Parks, 
1992). Students in the English as a Second Language (ESL) 
program at Seattle Central Community College develop portfolios 
that demonstrate listening, speaking, reading and writing skills 
(Loos, 1992). English, sociology, and art students at Washington's 
Edmonds Community College also develop portfolios to reflect 
learning in those courses (Martin, Newell, & Scarborough, 1992). 
Typically, the art portfolios have included materials using a variety 
of media; efforts to include multimedia information in the other 
content areas are being contemplated (Martin, 1992). Early positive 
results indicate that portfolios have application in a wide range of 
instructional areas. 



Demonstrations of Mastery 

Also called "exhibitions" and "presentations," demonstrations of 
mastery are more or less public performances intended to display 
competent knowledge and skill (McDonald, 1992). Demonstrations 
may be embedded in curricula and everyday practice and evaluated 
by ongoing, informal observations of students at work and over 
time. But more often, they are formal, culminating assessments 
occurring after a course of study. Demonstrations usually involve 
"production of discourse" (Newmann, 1992, p. 54) in which 
students summarize and integrate learnings in one or more 
disciplines. These may be supported by tangible products, 
experimental conclusions, or solutions to practical problems. 

For example, a demonstration of mastery in U.S. history at Walden 
III High School in Racine, Wisconsin, involves both written work 
and an oral presentation to a committee of two teachers, an adult 
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from the community, and an underclassman. Committee members 
assess 

1 . a research paper on a topic chosen by the student, 

2 . an oral elaboration of the paper, 

3. a time-line of events and movements highlighting points 
relevant to the student's research topic, and 

4. the student's response to questions and challenges. 

At Jefferson Community College in Louisville, Kentucky, students 
in all curricula are encouraged to participate in contests, complete 
projects, and display work in art exhibits. A file of qualitative 
evaluations of these efforts is kept on students who participate. 
These data contributes to an overall understanding of what the 
student has learned while attending the college. Efforts arc currently 
being made to collect these demonstrations in concrete form so that 
they may be presented in a portfolio (Koppel, 1991). 



Individual or 
collaborative 
investigations 
can lead to the 
development of 
expertise in 
specialized 
topics 



Projects 

Individually or collaboratively, students may be engaged in projects 
of various kinds. These activities may be original, interdisciplinary 
investigations that lead to the development of expertise in specialized 
topics and skills and, perhaps, new knowledge (Raven, 1992). 

For example, a student or a group of students might study a 
machine, a plant, or a transportation vehicle by reading, observing, 
building models, germinating seeds, interviewing, and/or recording 
progress. In the process, they might obtain information and develop 
skills that are specialized — and therefore hard to measure with a 
standardized test. But the knowledge and methodological 
competencies gained in such projects arc identifiable and assessable 
when students are observed in action and asked to explain what they 
have seen and learned. 



Performance Tasks 

Tasks that are intended to elicit student action and production and 
that directly reflect complex knowledge and skills are being 
developed and used to evaluate achievement in various content 
areas (Wiggins, 1991). Despite believing that these tasks are 
meaningful measures of important proficiencies, developers are 
nevertheless finding it difficult and time-consuming to construct 
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performance tasks in some content areas. Concern has been voiced 
that the specificity of tasks, combined with the time required to 
complete them, constrains the scope of what can be tested, raising 
alarm about performance tests' validity and reliability (Haertel, 
1992). 

In the U.S., performance tasks have been utilized at state and local 
levels. For example, the New Yoik State Department of Education 
was the first to mandate science performance tasks to be 
administered to fourth graders (Gould, 1992). Development of 
substitutably equivalent tasks, standardization of administration and 
evaluation, scoring rubrics, and calibration are foci of efforts to 
develop a national assessment system. These efforts are the subject 
of intense debate (see Smith et aL, 1992). 



Theoretical Bases 



Learning Theory 



Assumptions 

undergirding 

performance 

assessment 

parallel learning 

theory 



The assumptions undergirding performance assessment harmonize 
with learning theory advanced in cognitive psychology and 
qualitative research philosophy. To the extent that performance 
assessment permits or encourages unique responses to assessment 
tasks, each student's construction of reality and knowledge is 
honored. 

Implicit are perceptions such as 

• Learning and evaluation are subjective processes of the mind. 

• Learners are active constructors of knowledge rather than 
passive recipients. 

• Assessors are sensitive appliers of criteria and standards which 
may be largely intuitive. 

Learning is seen as a restructuring of the mind. This restructuring 
occurs as learners encounter new information, recognize it as 
important, and reconceptualize their knowledge to include that 
information. Expertise is gained as information is accumulated and 
as skill in using and applying information grows. Achievement is 
seen as the accumulation of increasingly sophisticated knowledge, 
ease of making connections among different kinds of knowledge, 
and skill in using and applying what is known (see Camp, 1990; 
Schuman, 1992; Shepard, 1991). 
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By way of contrast, consider some assumptions evident in 
standardized testing: 

• There is one right answer. 

• Fairness requires testees be treated identically. 

• Context variables (for example, what has actually been taught) 
should be discounted 

• Complex knowledge and skills can be meaningfully evaluated 
by means of multiple-choice questions. 

Achievement is the accretion and cataloging of isolated bits of 
knowledge. 

This view that knowledge is largely static and sequential and that 
learners are absorbers of identical knowledge structures fails to 
acknowledge the uniqueness of students and the uniqueness of each 
person's understandings (see Berlak, 1992a). 



Measurement Theory 



Validity and 
reliability are 
standards for 
determining the 
quality of an 
assessment 



Validity and reliability are crucial standards for determining the 
quality of an assessment. Very briefly, validity is a multifaceted 
concept of "the adequacy and appropriateness of inferences and 
actions based on test scores or other modes of assessment" 
(Messick, 1989, p. 13, emphasis in the original). The essential 
validity question is one of appropriateness of inference: What can 
be inferred about what students know and can do on the basis of 
their performances during assessment? 

Validity is notoriously difficult to estimate. Achievement is defined 
and determined in so many ways that it is impossible to say what 
exactly it encompasses, undermining of the appropriateness of 
inferences (Berlak, 1992a; Cole, 1988; Messick, 1989). In 
addition, inferences of achievement are easily confounded. For 
example, a student's general intelligence or "test-wiseness" can 
affect her test score and lead to erroneous estimations of 
achievement in a specific content area. 

Calculations based on reliability coefficients are sometimes used to 
estimate validity (Satterly, 1989). This is, at best, insufficient. 
Demonstrating die validity of an assessment practice or instrument 
requires gathering many kinds of evidence that inferences drawn, as 
reflected in the uses made of scores, are reasonable and appropriate 
without causing harm (Messick, 1989; Moss, 1992). 

Reliability traditionally refers to consistency of test scores (Feldt & 
Brennan, 1989). Reliability and validity are both necessary to 
determine whether a test accurately represents student ability or 
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knowledge. If a student attains consistent scores on parallel 
assessments, it strengthens inferences that this score represents what 
has been learned. But it is possible that consistent scores may be 
invalid. For example, a student may score 90% on several 
equivalent forms of a science test but the score may be more 
reflective of general intelligence, reading ability, or test-wiseness 
than it is of knowledge of science. In such a case, the tests have 
been reliable — the scores have been consistent — but the likely 
inference that the student is competent in science is of questionable 
validity. 

Different kinds of validity and reliability concerns arise with 
different kinds of student assessments. The following table 
suggests the nature of some of these concerns for both performance 
assessment and standardized testing. 



Validity and Reliability Concerns 



Performance assessment 



Validity Subjective, holistic evaluation opens possibilities of 
scores distorted by rater bias, level of expertise, 
familiarity with assessee, and so forth (Linn, 1991; 
Stiggins, 1991). 

Reliability Reliability can be difficult to obtain for two main 
reasons. First, defining reliability as consistency of 
scores for different performances is problematic 
because test-retest opportunities are limited by time 
and paucity of equivalent tasks (Linn, 1991; Linn, 
Baker, & Dunbar, 1991). Second, defining 
interrater reliability, a variant, as consistency of 
scores for a single performance from different 
assessors presents the difficulty that raters may 
disagree about the score for a single performance 
(see Stiggins, 1987). 



Standardized testing 

Validity Validity may be compromised by measuring 
achievement in complex domains via simplistic 
multiple-choice questions chosen for their capacity to 
differentiate and rank student performances and for 
their internal consistency (i.e., the correlation of 
performance on one test question with performance 
on all test questions) (Ennis, 1979) rather than for 
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content representativeness or relevance (Messick, 
1989). 

Reliability Reliability is compromised by test developers 1 
methods for calculating reliability coefficients — not 
from different scores attained on different 
administrations of equivalent tests or from different 
raters 9 evaluations of a single performance but rather 
from internal consistency of test items (Ennis, 1979; 
Satterly, 1989). Internal consistency as the driver 
for reliability coefficients presses for homogeneity of 
test content even in heterogeneous domains. 



Rater training 
should prepare 
instructors to 
form valid 
inferences 
based on 
evidence of 
student 
competence 



Because there is concern about variability in teacher scoring of 
student performances, training of assessors is advocated in many 
places (Linn, Baker, & Dunbar, 1991; Stiggins, 1987). This 
practice, it has been argued, increases the reliability of the 
assessment. However, training to increase reliability limits the 
range of inferences in order to reach consensus. In other words, 
training sacrifices validity to raise interrater reliability. Assessor 
training should focus not on restricting the range and bases of 
inferences instructors can make but rather prepares them to form 
valid inferences based on evidence of student competence weighed 
against reasonable, perhaps student-specific standards of 
performance. 



Advantages of Performance Assessment 



Student Outcomes 

Empirical evidence that performance assessment improves student 
outcomes is patchy. However, in both secondary and 
postsecondary education where performance assessment has gained 
a foothold among assessment practices, good news is reported. 

For example, at Walden HI High School, where graduation depends 
upon portfolios and demonstrations of mastery in sixteen academic 
and personal growth areas, alumni earn higher standardized test 
scores including ACT scores (Racine Unified School District, 
1990b), higher salaries in post-secondary employment (Racine 
Unified School District, 1990a), and higher collegiate freshman 
grade-point averages ("Grades," 1986; "Suburban Students," 1981) 
when compared to alumni from local high schools (Mabry, 1992). 

In the Pittsburgh Arts PROPEL schools, students creating portfolios 
that reflect the composition process reportedly write better, develop 
internal standards of quality, and think of themselves as professional 



118 

13C 



writers (Camp, 1990). Other sites also offer significant evidence of 
important and positive student outcomes (see Archbald & 
Newmann, 1992). 



Distinctions are 
bh red 
between 
assessment and 
learning when 
performance 
tasks are used 



Student Motivation 

A primary reason for using performance assessment is the 
opportunity for student participation in assessment processes, 
resulting in heightened motivation about school in general. In a 
Michigan pilot program, "Sites that had students actually begin 
developing portfolios reported increased [student] motivation, more 
interest in schoolwork, and increased self-esteem" (Sterner, Brown, 
& Smith, 1992). 

Advocates of performance assessment hold that students cannot 
fully display what they know and can do except in response to 
motivating tasks (Raven, 1992). Just as students learn better when 
they are motivated, they also perform better on assessments when 
they axe motivated. Interesting assessment tasks calling for student- 
valued and complex skills not only provide useful evaluations but 
also positive learning experiences. In this way, distinctions between 
assessment and learning are blurted; it is not merely that assessment 
informs future instruction but that assessment is instructive (Camp, 
1990). 



Self-assessment and Reflection 

Performance assessment often involves students in the evaluation 
process, eliciting reflection and self-assessment Self-assessment is 
a highly beneficial educational goal because the ability to monitor 
one's own performance is an important component of professional 
or adult competence. Although its attainment is obstructed by the 
remote scoring of standardized tests, development of self- 
assessment skill can be reasonably expected from many performance 
assessment techniques. Many performance assessment practices 
explicitly call for self-assessment 

Student reflection enhances personal investment in schooling by 
promoting appreciation and exploitation of opportunity for learning 
and career preparation. In this way, performance assessment helps 
students consider what they need to learn in order to become 
competent adults and professionals and motivates them to take 
advantage of opportunities to learn it 



Development of Personal Standards of Quality 

Implicit in the discussion above is that reflection and self-assessment 
promote the development of personal standards of quality. 
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Competence is distinguished by the ability to conceptualize and 
create quality products and services. Performance assessments that 
encourage or require formal or informal self-assessment provide 
opportunities to generate and adopt such standards. Reflection upon 
the adequacy of one's skills and performance is the crucial 
intermediate step in development of personal standards of quality. 
There is typically no provision for this step in traditional secondary 
and postsecondary curricula or timeframes (Raven, 1992). 



Implementation Issues 



Feasibility in Terms of Time, Funding, and Space 

There are many charges that standardized testing consumes more 
resources than results justify — approximately one-hundred hours 
per classroom per year (Smith & Rottenberg, 1991) and 
approximately half a billion dollars spent nationally per year (Paris, 
Lawton, Turner, & Roth, 1991). On the other hand, administering 
and assessing performance tasks of various kinds is — or will be — 
time-consuming (Linn, 1991), perhaps enormously so. 

Whether performance assessment requires greater funding than 
standardized testing is a somewhat controversial issue. Direct 
writing samples, currently incorporated into a number of large-scale 
testing programs, are more expensive to score than machine- 
scannable multiple-choice answers. But standardized testing also 
involves test construction, analysis, reporting of results, marketing, 
and so forth. Incorporating performance assessment into regular 
classroom activities might ultimately be less expensive, even when 
the cost of training teachers, hiring of external evaluators, setting up 
equipment, and providing storage space are considered. 



Preparation of 
instructors to 
carryout 
assessment of 
earning is 
crucial 



Faculty/Staff Development and Commitment 

In many performance assessment schemes, instructors are the 
primary assessors. However, the vast majority pf faculty members 
are not prepared to deal with the demands of holistic assessment of 
complex learnings and skills. In fact, most faculty are not prepared 
to deal with assessment tasks at all (Stiggins, 1991). For this 
reason, most performance assessment plans include recognition of 
the need for faculty development 

Performance assessment demands faculty preparation and also 
teacher commitment Faculty need to understand their task and also 
to believe in its benefits in order to commit the necessary time and 
energy (Raven, 1992). In some practices, additional external 
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expertise has been sought* For example, at Walden in High 
School, community members and students also act as assessors. 
Their intuitive understandings of what constitutes adult competence 
and their intimate acquaintance with students help them to make 
informed judgments. 



Public Credibility 

Performance assessment has not gained the public credibility 
awarded to standardized tests* Rightly or wrongly, people approve 
many uses of standardized tests, including uses known by 
measurement specialists and test developers to be ill-advised. The 
desire to compare educational performance — that of students, 
educational programs, colleges, districts, states, and countries — 
fuels demand for numerical indicators of quality to facilitate ranking. 
Unfortunately, these rankings are often made without considering 
crucial information regarding the context in which the tests were 
administered and the purpose they were intended to serve. 



Psychometric Vulnerability 



Validation of 
performance 
assessments is 
increasingly 
demanded 



It is doubtful that any state assessment officer or educational 
administrator would choose to administer to students a test whose 
validity or reliability were questioned; these psychometric standards 
are equated with assessment quality* An accusation that a test is not 
valid (although validity is really a property of the inferences based 
on a test, rather than a property of the test itself) or not reliable 
would be fatal 

Performance assessments are sometimes administered without any 
empirical evidence of validity and reliability* Although some 
evidence exists (see Mabry, 1992), claims of the validity and 
reliability of these types of measures tend to be speculative (Moss, 
1992). Pressure is increasing for these techniques to demonstrate 
validity and reliability or, alternatively, for researchers to develop 
new conceptions of validity and reliability or new validation 
criteria more appropriate to these types of assessments (Frederikson 
& Collins, 1989; Linn, Baker, & Dunbar, 1991; Moss, 1992). 
Now that exploration of performance assessment techniques is 
becoming widespread, demands for evidence of success will likely 
increase (Brewer, 1992). 



Legal Standing 

To date, no federal court cases have challenged performance 
assessment, but legal precedent regarding standardized testing of 
students and performance testing of employees suggest that they 
involve potential legal issues. If an assessment has adverse impact 
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on a protected minority group, validity is a probable concern. 
Questions such as the following would likely be posed: 

♦ What constitutes evidence of validity? 

♦ Must the assessor prove the assessment is valid or must the 
student being assessed prove it is invalid? 

♦ Will evidence of more valid alternatives be required? 

Many educational decisions are of the high-stakes variety — that is, 
they strongly affect people or institutions. Assessment data are 
often used to justify high-stakes decisions. For example, 
standardized test scores are regularly used as the basis for selecting 
students for college enrollment cr for special services such as gifted 
or remedial programming and for selecting schools or districts for 
rewards or corrective action. 

If an assessment is used for high-stakes decision making, courts are 
likely to require high interrater reliability by disinterested raters — 
forcing teachers out of assessor roles. However, the courts are also 
likely to allow good faith attempts to advance measurement practice 
(Phillips, in preparation). 



Suitability for Large-Scale Assessment or Accountability 

Considerable question exists as to the appropriate purposes and uses 
for performance assessment One concern is whether performance 
assessment suitable for classroom and/or school or college uses may 
also be suitable for district, state, or national assessments. Some 
argue that these techniques are ipsative 2 measures suitable for 
judging the achievement of individual students, but that they fail as 
comparative measures forjudging individual or group achievement; 
others counter that scores can be made comparable by means of 
moderation 3 (Radnor, 1991; Vermont State Department of 
Education, undated) or rater training (Stiggins, 1991). 

That performance assessments are context-specific (Berlak, 1992b) 
is seen by some to be an advantage. Sensitivity to local contexts 
enhances understanding of student achievement within that context, 
but it obstructs comparison of students in different contexts. For 
this reason, doubts exist as to whether performance assessment is 
viable for an entire district, much less for a state or for the country. 



2 Ipsative measures rely on the assessee as the sole referent, evaluating* for instance, the relative strengths 
and weaknesses of an individual or her progress over time. Contrastingly, comparative measures facilitate 
judgments about an individual's attainment compared to that of a group of apparently similar others. 

3 Moderation is a process of developing agreement as to performance criteria, standards, and evaluations. It 
is characterized by discussion among assessors as to what to look for in student performance, what 
constitutes sufficient competence, and what scores to award to particular student artifacts. 
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Large-scale performance assessments are being implemented (see 
Baker, Aschbacher, Niemi, & Yamaguchi, 1992), suggesting the 
possibility of neutralizing the effect of local contexts for cross- 
context comparisons. But if sensitivity to context is perceived as a 
strength, diluting that sensitivity may be a serious mistake. 

Large-scale assessment programs usually involve high-stakes 
comparisons of students, personnel, and/or educational institutions 
or entities. High stakes to institutions invariably corrupts the 
assessment of individuals (Linn, 1992). This is because low 
student scores can embarrass education providers, who are in a 
position to manipulate scores in a variety of ways (Cannell, 1987). 
For example, 

Test time can be lengthened by the person administering the 
test. 

Teachers can be shown tests early and drill students on test 
problems. 

Hints or too much help can be given in answering questions of 
students taking tests. 

Special education students can be sent on field trips on test days 
or defined as being ineligible for testing. 

Curriculum can be manipulated to ensure that test-specific 
topics are taught before tests ate given. 

"Score pollution" (Haladyna, Nolen, & Haas, 1991) deprives 
scores of meaning. Meaningless scores obstruct student, personnel, 
program, and institution evaluations. This inevitable situation, 
evident across the country, makes the use of test scores for 
accountability purposes an exercise in delusion. 
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Conclusion 



This book has discussed innovations in outcomes assessment for 
postsecondary education identified during a two-year research 
effort of the University of Illinois site of the National Center for 
Research in Vocational Education (NCRVE). This project 
involved two-year public and private postsecondary institutions in 
a descriptive research study to identify and document various 
approaches to outcomes assessment. A primary goal of this 
publication was to describe these approaches and to challenge 
readers to consider how they could be applied to their own settings 
to stimulate improvements in vocational education. Local leaders, 
institutional researchers, and policymakers were our primary 
audience. 

As was pointed out in Chapter One, there are many driving forces 
behind efforts to improve outcomes assessment in postsecondary 
institutions. Forces external to the colleges — such as government 
agencies, accreditation associations, business and industry, and 
new federal mandates — all encourage improved outcomes 
assessment practices. Each of these groups has a somewhat unique 
rationale for encouraging outcomes assessment; however at some 
level, each recommends assessment as a means of enhancing 
student learning. The need to improve programs and enhance 
assessment processes is also recognized internally. The increased 
diversity in student populations and in patterns of enrollment are 
behind die efforts of many postsecondary institutions. 

Postsecondary institutions are being encouraged by many groups to 
seek a broader set of outcomes than they have assessed 
traditionally and reexamine the means by which these outcomes are 
being measured. As new initiatives such as Tech Prep (technical 
preparation) and academic and vocational integration emerge, 
alternative approaches to outcomes assessment are essential. This 
book has sampled a selected few of these alternative approaches 
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and suggested ways each could contribute to improving 
postsecondary education. 



Six Alternative Approaches 



Total Quality 
Management 
(TQM) 



The Student 
Success Model 



In this book, six different approaches to outcomes assessment were 
described. Each of these approaches provides a unique perspective 
to conceptualizing, collecting, and analyzing outcomes for 
postsecondary education. A brief summary of these alternative 
approaches helps to provide a framework for thinking about issues 
and strategies related to implementation of outcomes assessment. 

Business and industry in America is adopting total quality 
management (TQM) because of its need to improve quality to 
increase economic competitiveness. First advocated by Deming 
in Japan TQM is now being contemplated in relation to educational 
reform. Fundamental goals of TQM, reported in Chapter Two, are 
to manage and improve work processes; identify, meet and exceed 
customer needs; ensure that the entire system is the focus of 
measurement for the purpose of making continuous improvements; 
and ensure that everyone is involved in carrying out the initiative. 
In a TQM approach, outcomes assessment is viewed as evidence of 
the effectiveness and efficiency of processes and programs 
interacting within the institution. Outcome measures provide 
evidence of the value of vocational education to students; 
deviations from specified performance standards are analyzed and 
fed back into the system to make improvements. Assessment for 
the purpose of accountability is recognized as part of this approach, 
but it is not the primary goal. The more critical purpose of TQM is 
to create and support an environment that yields continuous quality 
improvement. 

The goal of the student success model is to create an institutional 
environment that contributes to developing and maintaining student 
success. The concept of "success" is defined by students, many of 
whom enroll in vocational education programs, as they enter and 
move through the postsecondary institution. This model is based 
on the premise that students are in the best position to define 
"success" for themselves and, therefore, for the institution and its 
programs. Student goals and intended outcomes form the basis for 
an extensive, institution-wide assessment system which potentially 
provides a far different approach than an assessment system 
devised from outcome measures and performance standards 
established by external groups (e.g., governing boards and 
accreditation agencies). In practice, when using the student 
success model, Santa Fe Community College has found it feasible 
to meet its own high standards, as was reported in Chapter Three, 
as well as to demonstrate accountability to external bodies. An 
important part of this model involves encouraging students to set 
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high standards, documenting their goals and intended outcomes, 
and monitoring their progress toward obtaining these outcomes. 



Value-added 
Assessment 



Concept 
Mapping 



Assessment of 
vocational 
education 
outcomes 



The concept of value added can be applied to assessment of 
institutional effectiveness, program quality, or student learning. In 
Chapter Four, how value-added assessment can be used to 
determine the value of an educational experience for students was 
explored At the heart of value-added assessment is a strategy that 
provides a baseline of information about entry-level competency, 
often determined with a pretest, compared with exit-level 
performance, usually measured with a posttest. Through this 
design, it is purported that an institution or program can determine 
what value it has added to a student's array of competencies. By 
using this approach in vocational education, it would be possible to 
determine skills attained by students as they complete critical 
levels (i.e., benchmarks) of a program which could be helpful in 
determining how effective a program has been for the many 
individuals who move in and out of college or into the job market 
without finishing a degree or certificate. Further, even though 
value-added assessment has been conducted traditionally using a 
pretest and posttest design, the potential for using alternative 
assessment approaches such as performance assessment appears 
feasible. 

Concept mapping is a practical approach to conceptualizing how an 
educational program works as a basis for identifying outcomes. As 
was described in Chapter Five, it is a bottom-up approach that 
relies on input from practitioner stakeholders who play a part in the 
design and delivery of a program. The results of a concept 
mapping activity are displayed in a pictorial map that shows what 
practitioners think about how a program produces outcomes. 
Concept mapping employs small group processes sorting and 
rating techniques, and multivariate statistical analysis techniques to 
visually represent the program in the ibrm of a map. Vocational 
educators can use the approach to conceptualize outcomes by 
involving postsecondary educators, administrators, and students as 
well as community, business, industry, labor, or other 
representatives. Information produced by a concept mapping 
activity implicitly and explicitly defines what vocational education 
experts believe in and what they practice. The maps have been 
found to be useful for conceptualizing vocational education 
outcomes in three different postsecondary education settings. 

A comprehensive approach to outcomes assessment for vocational 
education must consider the needs of its clients, the educational 
processes associated with its programs, and the outcomes achieved 
by its student. Developing a set of outcomes without 
thorough understanding of client needs or the goals of educational 
processes is potentially a meaningless exercise. Chapter Six 
suggested that a comprehensive set of outcomes should evolve 
from the economic, educational, and psychological and 
sociological underpinnings of vocational education. The process of 
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designing outcomes assessment for vocational education involves 
five steps: (1) selecting outcomes to be assessed, (2) identifying 
indicators for each outcome, (3) identifying information collection 
arrangements, (4) determining the analysis procedures, and (5) 
reporting the information. 

Performance Performance assessment is an umbrella term used to describe 

Assessment techniques such as "alternative" or "authentic" assessment. Each of 

these assessment techniques is designed to assess directly what 
students know and can do in a form other than paper and pencil or 
multiple choice tests. Several examples of these techniques were 
described in Chapter Seven. They are profiles and records of 
achievement, portfolios, demonstrations of mastery, projects, and 
performance tasks. These types of techniqueg^are particularly 
advantageous in measuring student outcomes that are difficult to 
quantify using paper and pencil or multiple choice tests and in 
motivating students to learn, involving them in evaluation 
processes and promoting their development of personal standards 
of quality. Since many of these techniques are being used only 
recently, educators continue to struggle with such issues as cost, 
faculty development, and public credibility. Although vocational 
educators have been active users of performance tasks historically, 
the recent development of other assessment techniques such as 
portfolios or curriculum resumes appear to hold promise new 
advancements for the field. 



Twelve Implementation Strategies 



There are a number of reoccurring themes that appear throughout 
this book. Two of these are the importance of linking outcomes 
assessment to institutional mission and the necessity for 
understanding the needs of students as a basis for establishing 
outcomes measures. In this section, we present these and ten 
additional implementation strategies as a basis for encouraging 
implementation of outcomes assessment. 

Strategy 1: Ensure that a clear institutional mission provides a 
framework for assessing vocational education, including 
institutional outcomes, program outcomes, and student 
outcomes. Knowing exactly what drives the institution and what is 
envisioned to be accomplished as a collective body is crucial to 
developing meaningful outcomes assessment systems for all 
educational curriculum, including vocational education. 

Strategy 2: Focus outcomes assessment on improving the 
entire institution (i.e., system) and the processes functioning 
within it* To accomplish this, it is necessary to examine the 
adequacy of inputs, how inputs link to processes, and how 
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processes ultimately influence outcomes. Clearly, this approach 
requires that vocational education not be isolated from other 
curriculum efforts of the institution. In fact, it is essential to know 
how other programs and services within the institution influence 
the vocational education curriculum and its students. 

Strategy 3: Gain top institutional commitment to an outcomes 
assessment system. Chief executive officers (CEOs) and other 
institutional leaders must commit material and personnel resources, 
build a shared vision, and devise policy that supports faculty and 
staff in carrying out outcomes assessment for the purposes of 
improving quality and demonstrating accountability. 

Strategy 4: Provide mechanisms for scanning the external 
environment and develop processes to enhance the reciprocal 
relationship between colleges and their local communities. 

Community colleges and other two-year postsecondary institutions 
cannot operate in a vacuum. It is crucial that data collection 
procedures be developed and carried out routinely to inform both 
the college and the surrounding community about needs as well as 
services that each can provide. This exchange of information 
between internal and external stakeholders (i.e., customers) is 
critical to maintaining an outcomes assessment approach that can 
feed information back into the system to make improvements. 

Strategy 5: Methods used to conceptualize outcomes for 
vocational education should be democratic, actively involving a 
wide range of stakeholders. Consensus is needed from the 
stakeholders of vocational education, those internal and external to 
the institution, about outcomes that should be achieved. Once a 
broad-based, grass-roots consensus is reached, these outcomes can 
be translated into outcome measures, performance indicators, and 
standards. 

Strategy 6: Make the goal of identifying and meeting 
individual student needs a top priority for vocational 
education* A philosophical perspective that places individual 
student success at the center of the program create c the necessity 
for a highly flexible assessment system. Whereas this system will 
be complex to operationalize, it promises to yield valid information 
about the effectiveness of vocational education programs, 
especially compared to systems based on goals and outcomes that 
have little relevance to students or faculty in postsecondary 
education. 

Strategy 7: Build an organizational structure that supports 
data-driven decision making, planning, and program 
improvement At the heart of this structure are teams comprised 
of faculty, staff, and possibly students. These groups are 
empowered to make many important decisions about 
implementation of outcomes assessment. For this structure to work 
effectively, there must be clear lines of communication throughout 
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the institution. In addition, people need opportunities to acquire 
the skills and knowledge needed to make decisions about outcomes 
assessment 

Strategy 8: Use multiple measures to develop comprehensive 
records of student performance. These records need to show 
how students axe performing as/they progress through a vocational 
education curriculum as well as how they are performing at the 
time of completion. Multiple 'measures, using a mixed-method 
approach that captures qualitative and quantitative data, appears to 
provide valuable information about the different kinds of student 
learning that occur in the postsecondary setting. 

Strategy 9: Conduct program audits on a routine basis. 
Whereas there is value in conducting program reviews on a regular 
schedule, typically every five years, this approach has potential for 
missing opportunities for improving programs on a continuous 
basis. A five-year funding cycle assumes a stable environment, 
however much change can occur in a five-year time period. In 
addition to a regularly scheduled five-year review, we recommend 
using program audits to eliminate problems as they arise. In 
essence, data should be collected and analyzed continuously on key 
aspects of any vocational education program. These audits provide 
information for improvement on an on-going basis. 

Strategy 10: Make sure there are feedback loops for 
continuous improvement. A systematic approach to conducting 
outcomes assessment at the institution, program, and soident level 
helps to ensure that information is used to make improvements. 
Opportunities to improve are identified and addressed when an 
organizational structure is employed that provides open lines of 
communication about how the institution is functioning. 

Strategy 11: Design reward systems (e.g., compensation, 
promotion) that support the use of outcomes assessment system 
for improvement. It is essential that an institution revise reward 
policies to reinforce individual as well as collective, team efforts to 
improve programs. To be effective, this type of system needs to be 
highly responsive to needs; it must also clearly emphasize the 
importance of improving the entire system to enhance outcomes. 

Strategy 12: Develop dissemination strategies that support the 
intended use of the data. Sometimes these dissemination 
strategies involve distributing findings widely; other times they 
mean limited circulation of information. The important deciding 
factor is made by individuals who need to use the information to 
make improvements. 
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Implementation Issues 



We conclude this book with several important questions about 
outcomes assessment in postsecondary education. These questions 
are posed to encourage innovation, not to stifle progress. Solutions 
to these questions will not be found quickly or easily. However, 
they are posed to stimulate discussion in your institution about how 
alternative approaches to outcomes assessment can be enhanced to 
improve the quality of vocational education. 



Discussion Questions 



How can your institution develop a culture that values outcomes 
assessment? Is it possible to move away from the perception that 
evaluation is to be dreaded rather than appreciated for how it can 
improve quality? 

What should be the balance between outcomes assessment for 
accountability and program improvement? If your assessment 
system is designed to meet the mandates of external groups, how 
can it also meet your internal needs for policy improvement? 

How can commitment from your administrators and faculty be 
gained and reinforced to sustain an outcomes assessment system 
over the long term? 

Who should be involved in the process of conceptualizing 
outcomes? What role should students play? How will your 
institution resolve competing interests of different stakeholder 
groups? 

How should your institution move forward with assessing 
outcomes that all agree are important but are extremely difficult or 
costly to measure? 

How can the next generation of performance assessment 
techniques be developed and implemented more quickly within 
your institution? 

How can your institution ensure that its outcome measures will not 
isolate vocational education rather than integrate it into the 
curriculum? 

How can the concept of continuous improvement be employed for 
your postsecondary vocational education programs? What kind of 
information will be needed to accomplish this goal? 
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